Adding json format for import #766

mathispernias · 2025-05-20T18:08:22Z

Tests pass
ruff format
README.md updated (if relevant)
CHANGELOG.md entry added

jochenchrist · 2025-05-20T19:07:24Z

CHANGELOG.md

@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [0.10.25] - 2025-05-07

 ### Added
+- `datacontract import --format json`: Import from JSON files


Please move this to

Unreleased

jochenchrist · 2025-05-20T19:09:20Z

mermaid2.html

Does not belong here, does it?

my mistake i removed this file

jochenchrist · 2025-05-20T19:09:28Z

query

I removed it!

jochenchrist · 2025-05-20T19:11:00Z

tests/datacontract.yaml

@@ -0,0 +1,216 @@
+dataContractSpecification: 1.1.0


Is this datacontract.yaml used?
Why has it such as generic name?

Sorry I forgot to remove it, it's done!

jochenchrist · 2025-05-20T19:12:07Z

tests/fixtures/import/product_detail.json

@@ -0,0 +1,5002 @@
+{


please move these files to a subdirectory fixtures/import/json

okay it's done

jochenchrist · 2025-05-20T19:12:46Z

tests/fixtures/import/product_detail.yaml

@@ -0,0 +1,2301 @@
+dataContractSpecification: 1.1.0


please rename to product_detail.datacontract.yaml

I renamed it

jochenchrist · 2025-05-20T19:13:16Z

tests/fixtures/import/productsimple-datacontract.yaml

@@ -0,0 +1,317 @@
+dataContractSpecification: 1.1.0


please rename to productsimple.datacontract.yaml

I renamed it

jochenchrist · 2025-05-20T19:20:00Z

This implementation looks like it supports JSON files with single structures.
In practive, we often see JSON files with new line delimited entries that contain 1000 messages.

Example: https://github.com/datacontract/datacontract-cli/blob/main/tests/fixtures/s3-json/data/inventory/year%3D2022/month%3D04/day%3D20/hour%3D00/inventory%2B0%2B0001327496.json

Can you please make sure that the JSON importer also supports JSON documents with multiple entries?

…ntract-cli into json_import

mathispernias · 2025-05-22T13:03:25Z

The JSON importer can now also takes NDJSON (Newline-delimited JSON).

simonharrer · 2025-05-28T09:08:45Z

I like it. Can you make the test jsons a little smaller? And make sure the test jsons and their imported contracts have the same file prefix.

mathispernias · 2025-05-28T18:58:27Z

I updated the JSON tests to include three cases—simple JSON, complex JSON, and NDJSON— I also changed the file prefix.

jochenchrist · 2025-05-29T05:07:15Z

datacontract/export/odps_export.py

Empty file?

jochenchrist

LGTM, thanks for your contribution

mathispernias added 20 commits May 12, 2025 19:22

adding json file with nested structure

191a957

adding json_importer

ccf769a

correcting nested problem

8f305e6

adding comment

c32ac77

adding comment

b8fdf93

deleting unecessary lines

c07dd7c

adding comments

e096398

adding comments

8a7b95f

adding comments

b46b7c0

deleting tests files

cedd475

adding comments

79d42b3

changing comments

587a42d

adding tests

b00c8ee

adding test for import json

c89e20a

changing readme

43ccae4

adding tests for import json

e25987b

trying solve test errors for json import

8a1eefe

correcting error when datacontract test on json import almost done

1e6d423

fixing bug

2d4a23b

ruff format and updating changelog

aa0826a

mathispernias changed the title ~~Json import~~ Adding json format for import May 20, 2025

jochenchrist reviewed May 20, 2025

View reviewed changes

moving log to unreleased

53bd4ff

mathispernias added 13 commits May 20, 2025 15:55

cleaning files and adding multi entries

062983b

ruff format

e7981c8

moving test files

6ca8ef3

Merge branch 'main' into json_import

389db35

rename test file

01210b3

Merge branch 'json_import' of https://github.com/mathispernias/dataco…

2380839

…ntract-cli into json_import

multi lines json

ffa14e0

ndjson implemtation

28eb6f2

changing comments

add222b

ndjson implementation

a2bc12a

updating tests

db34950

updating tests

b32472c

adding test for ndjson

de2720b

ruff fornat

920d99b

mathispernias added 2 commits May 28, 2025 14:40

changing file prefix + changing tests json import

d1c1504

changing file prefix + changing tests json import

8b00e82

jochenchrist reviewed May 29, 2025

View reviewed changes

datacontract/export/odps_export.py Outdated

Copy link

Contributor

jochenchrist May 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Empty file?

jochenchrist approved these changes May 29, 2025

View reviewed changes

deleting empty file

1c5cdc2

Adding json format for import #766

Are you sure you want to change the base?

Adding json format for import #766

Uh oh!

Conversation

mathispernias commented May 20, 2025

Uh oh!

Choose a reason for hiding this comment

Unreleased

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jochenchrist commented May 20, 2025

Uh oh!

mathispernias commented May 22, 2025

Uh oh!

simonharrer commented May 28, 2025

Uh oh!

mathispernias commented May 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jochenchrist left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!