fix default_paramstyle #27
Conversation
Codecov Report
@@ Coverage Diff @@
## master #27 +/- ##
=====================================
Coverage 100% 100%
=====================================
Files 4 4
Lines 393 391 -2
Branches 48 48
=====================================
- Hits 393 391 -2
Continue to review full report at Codecov.
|
|
with 2267650 I still receive the pydantic error |
|
I just fixed it. it work from sqlalchemy.orm import sessionmaker
Session = sessionmaker()
Session.configure(bind=engine)
s = Session()
s.bulk_save_objects([Pets(name='pet_name1'), Pets(name='pet_name2')])
s.commit()output 2019-10-16 04:20:19,575 INFO sqlalchemy.engine.base.Engine select version()
2019-10-16 04:20:19,575 INFO sqlalchemy.engine.base.Engine {}
2019-10-16 04:20:19,588 INFO sqlalchemy.engine.base.Engine select current_schema()
2019-10-16 04:20:19,588 INFO sqlalchemy.engine.base.Engine {}
2019-10-16 04:20:19,603 INFO sqlalchemy.engine.base.Engine SELECT CAST('test plain returns' AS VARCHAR(60)) AS anon_1
2019-10-16 04:20:19,603 INFO sqlalchemy.engine.base.Engine {}
2019-10-16 04:20:19,611 INFO sqlalchemy.engine.base.Engine SELECT CAST('test unicode returns' AS VARCHAR(60)) AS anon_1
2019-10-16 04:20:19,611 INFO sqlalchemy.engine.base.Engine {}
2019-10-16 04:20:19,618 INFO sqlalchemy.engine.base.Engine show standard_conforming_strings
2019-10-16 04:20:19,618 INFO sqlalchemy.engine.base.Engine {}
2019-10-16 04:20:19,642 INFO sqlalchemy.engine.base.Engine BEGIN (implicit)
2019-10-16 04:20:19,654 INFO sqlalchemy.engine.base.Engine INSERT INTO pets (name) VALUES (:name)
2019-10-16 04:20:19,654 INFO sqlalchemy.engine.base.Engine ({'name': 'pet_name1'}, {'name': 'pet_name2'})
2019-10-16 04:20:19,665 INFO sqlalchemy.engine.base.Engine COMMIT
Process finished with exit code 0 |
|
@Rubyj |
|
@koxudaxi I still receive an error but it is unrelated: Is there a way for me to increase this limit? Or will this bulk insert take too long. |
|
I found it in https://docs.aws.amazon.com/rdsdataservice/latest/APIReference/API_BatchExecuteStatement.html I think the limitation can not be changed. We can ask about it to AWS supports... |
Yeah, I just saw that as well. Do you have any ideas for workaround? Build multiple lists each being 1000 long maybe? |
@Rubyj |
I have about 60k objects to bulk insert. In theory, I can make 60 lists each with 1000 objects and bulk insert them all. |
|
We have another way that the library divide objects each 1000. And the library call operations. |
I will try to divide my data by 1000. It should not be too hard. |
|
@koxudaxi I was able to batch my data by 1000. However, now I run into the following error. This happens on the model with a relationship to the other model |
|
@Rubyj |
I set |
|
@Rubyj |
Ok! I will continue to debug today and post what I find for you to see tomorrow. I can run outside of chalice using regular python. Thank you!! Even outside of |
|
It works to run pydataapi(mysql) with sam result: ResultProxy = engine.execute("select * from pets")output is
|
Weird. I wonder why I can not get it to work in my @koxudaxi I took a look at the data that is being sent to the data api from python. It looks like the foreign key which i showed above is not included in the the data that is being sent to Here is an example: I do not see the reference back to the Assigning the parent object like I did here in my code does not seem to make it to the We could start with something like this: |
|
I have test simple relation tables. It works fine. However, I have run with Aurora Serverless MySQL. class Parent(base):
__tablename__ = 'parent'
id = Column(Integer, primary_key=True)
name = Column(String(255, collation='utf8_unicode_ci'), default=None)
class Child(base):
__tablename__ = 'child'
id = Column(Integer, primary_key=True)
name = Column(String(255, collation='utf8_unicode_ci'), default=None)
parent_id = Column(Integer, ForeignKey('parent.id'))
parent = relationship('Parent', backref='childs')
s = Session()
parent_object = Parent(id=1, name="test")
child_object = Child(id=1, parent=parent_object)
s.bulk_save_objects([parent_object])
s.bulk_save_objects([child_object])
...
I think SQLAlchemy split SQL to parent and child Also, I'm interested in the value.
|
|
@Rubyj If you can show me DDL for creating table and SQLAlchemy's Table classes then, I can try it. |
This is strange. I am not sure why it is described as
Yes. Here they are: What is the best way to get DDL from serverless? I took a look at the table information and it matched these models:
|
|
@Rubyj
Sorry, I don't know.
It may be bugs 😖 PS. I just created tables from SQLAlchemy classes. |
Yes, I created the tables using alembic. After creating models.
alembic.ini alembic/env.py - edit run_migrations_online() to use create_engine
|
|
alembic dump an error. I create all the tables in a simple way. It works. I try to build-insert I run the lines result: ResultProxy = engine.execute(Select([Variant]))
print(result.fetchall())output [(2, True, True, True, True, 'test', True)] |
Not sure about that error. What does your alembic setup look like? Ok I will also try to look at my models and make length longer. May have error there. |
why are there so many boolean values? Hmm maybe this is a problem with my data then. I will need to investigate further why this thinks my |
|
It's a bug!! Real Values are Its should be |
aha! Although, I think this is a different bug 😅 I wonder if this is related to my |
|
@koxudaxi I tried increasing the length on my models but I still get the |
|
@Rubyj |
Yes, I can. I have been trying to set breakpoint, but I do not know the correct place to do so to get a good error message. |
|
Here !! for checking request of batch_execute |
|
@koxudaxi I got the logging to work!!! I think nan is indeed the issue. I see this:
|
|
Should clean up data by pydataapi? We should fix the converting method. def convert_value(value: Any) -> Dict[str, Any]:
if isinstance(value, bool):
return {'booleanValue': value}
elif isinstance(value, str):
return {'stringValue': value}
elif isinstance(value, int):
return {'longValue': value}
elif isinstance(value, float):
return {'doubleValue': value}
elif isinstance(value, bytes):
return {'blobValue': value}
elif value is None:
return {'isNull': True}
else:
raise Exception(f'unsupported type {type(value)}: {value} ') |
It is working now!!! But It is taking a little while because I have around 100,000 models to insert. The problem was I had Sorry for the trouble! |
|
@Rubyj |
|
I think it is OK for user to handle this special vale. Otherwise, numpy becomes a dependency of pydataapi. My inserts have been running for over 5 minutes 🤦♂️ |
|
@Rubyj We could fix a lot of bugs in this PR 🎉 |
|
@koxudaxi Ofc ourse!! Thank YOU! One last thing. Do you know if |
|
Sorry, I don't know it. However, It works |
Ok I am interested to see if it works for relationship. I can look. |
|
I have released version 0.4.4 🎉 |
The PR fixes
default_paramstylefor dialect.Related Issue
#19