Repository navigation
Converting SPDE to parquet via sd2pq #624
|
Hi I'm looking to convert some very large legacy I saw I was not able to locate entries in the documentation and examples on how to connect with the In short:
Thank you in advance! |
Answered by
tomweber-sas
Dec 6, 2024
Replies: 10 comments 6 replies
|
Yes, in SAS you submit a libname statement to connect to a datasource, be it SAS data sets, SPDS/SPDE data, DBMS data ... Then, hopefully the sd2pq() method which was user contributed, will work for you! |
0 replies
Answer selected by
TPDeramus
|
Thank you! Will give it a shot here in a moment. Unrelated, the link appears dead for me. |
0 replies
|
Don't know why that would be. Just google |
0 replies
|
Have had to delay the implementation due to a few things so will update you as soon as I'm able. Thank you for your help with this. |
0 replies
|
Is there anyway to get a quick "peek" at the generated I suspect I'm calling the wrong table and what I'm grabbing is empty but I can't seem to be certain: |
1 reply
Answered my own question. And accessing the metadata seems to do the trick. |
|
So this worked quite swimmingly. Thank you so much! That said, could you recommend a way to I've tried a few suggested attempts and they usually result in bad calls on the If nothing immediately comes to mind, happy to go ahead and resolve and close the question. |
2 replies
|
Well, there's a couple thoughts. I didn't write this method (sd2pq), a user contributed it, so I'm not up on it 100% but there a lot of options they included, including partitioning of the data set so it's in multiple parts (individual files). I'm not sure if that would let you just reference parts of it after the fact from python, or not. The other thought would be only pulling part of the SAS Data Set over, which may actually be what you're asking. Using data set options to subset it (obs= and firstobs=) is a very easy way to break it up into smaller parts. You can use the DSOPTS attribute of the SASdata object to accomplish that. See dsopts here: https://sassoftware.github.io/saspy/api.html#saspy.sasdata.SASdata |
|
That would be what I'm asking, I just couldn't quite get the call to I think the Thank you once again and appreciate you developing such a useful package! |
|
Here's an example of just providing dsopt on the sd2df call so you can easily subset rows. You can set it on the SASdata object and change it between calls too. There's more than one way depending upon if you whant it associated to the SASdata object for all of the methods you use, or just provide it on a method for a one time use. FYI, the cars data set has 428 rows. obs= is a bit of a misnomer, it should have been called lastobs=. |
0 replies
|
as a comparison, setting dsopts on the SASdata object (obs 1-10), then applies that to all methods: |
0 replies
|
Thanks for clarifying that! |
0 replies
|
Hi @tomweber-sas . Re-opening this again if I may. I seem to be getting an error with "failed" chunks for some of the sas9's I'm trying to convert. I created the following workflow to do it: And usually pass it in the following manner via bash commands: It's worked for several of our datasets. However, for particularly large one, it keeps throwing some form of "chunk" failure that gives me the impression the chunk size is too big or too small in some way at a certain point. with a I initially thought I was just running the same scripts on accident but it's been occurring far too frequently even with the extra checks. environment.yml is below if you require it: |
3 replies
|
Actually scratch that, pretty sure this is what's causing the error: And that looks like a bad date casting (which is a different issue) |
|
Sorry not to get back, been heads down on some internal SAS 9 issue I have to address. I didn't write this method, it was contributed by a user. That said, I did help with it and did test out the basic functionality before releasing it. However, there are lots of options and multiple different ways to use this. I didn't vet every path; wouldn't even know how to. This error appears to be with data/time/datetime values coming over from SAS. The most obvious thought is maybe this code path doesn't handle missing values, or values out of range of numpy. SASPy handles all of that for sd2df(), but I see the code that's having issue here, is specific to this method, and may have an isue that needs to be addressed. Have you narrowed down the data that's causing this to see if it's caused by missing or our of range values? |
|
Apologies for the delay on this. I haven't been able to narrow it down because the dataset is so large (which is its own issue), but I can confirm 2 things.
I will update if I discover anything else, but thank you for the suggestion on the missing values! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Yes, in SAS you submit a libname statement to connect to a datasource, be it SAS data sets, SPDS/SPDE data, DBMS data ...
Then you have a libref, which references that data, and then the table names are the names of the data sets/tables.
Here's the doc for libname SPDE
Then, hopefully the sd2pq() method which was user contributed, will work for you!