Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(api-v2): Add support for PDF files #1206

Merged
merged 101 commits into from
Nov 28, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
ff19529
feature (api-v2): Upload non-image files (ongoing).
Feb 6, 2019
91bd6f3
Merge branch 'develop' into wip/1202-upload-non-images
Mar 1, 2019
6574df4
Merge branch 'develop' into wip/1202-upload-non-images
Mar 6, 2019
2a6012d
Merge branch 'develop' into wip/1202-upload-non-images
Mar 20, 2019
7a554f1
Merge branch 'develop' into wip/1202-upload-non-images
Mar 25, 2019
0b90fcf
Merge branch 'develop' into wip/1202-upload-non-images
Mar 25, 2019
822936d
Merge branch 'develop' into wip/1202-upload-non-images
Mar 26, 2019
e48d107
test(api-v2): Fix test.
Mar 26, 2019
db04659
test(api-v2): Update test.
Mar 26, 2019
d6a9965
Merge branch 'develop' into wip/1202-upload-non-images
Mar 27, 2019
db04d6e
Merge branch 'develop' into wip/1202-upload-non-images
Mar 28, 2019
9c6b43d
Merge branch 'develop' into wip/1202-upload-non-images
Mar 28, 2019
0e2ae67
Merge branch 'develop' into wip/1202-upload-non-images
Apr 1, 2019
321eb7f
Merge branch 'develop' into wip/1202-upload-non-images
Apr 2, 2019
d1a8dee
Merge branch 'develop' into wip/1202-upload-non-images
Apr 8, 2019
67bc622
Merge branch 'develop' into wip/1202-upload-non-images
Apr 11, 2019
63d2e1d
Merge branch 'develop' into wip/1202-upload-non-images
Apr 11, 2019
5b0edec
Merge branch 'develop' into wip/1202-upload-non-images
Apr 15, 2019
2d490df
Merge branch 'develop' into wip/1202-upload-non-images
Apr 15, 2019
189a792
Merge branch 'develop' into wip/1202-upload-non-images
Apr 18, 2019
dfaaf83
Merge branch 'develop' into wip/1202-upload-non-images
May 23, 2019
e8c51ec
Merge branch 'develop' into wip/1202-upload-non-images
May 24, 2019
049498e
test: Fix compile error in test.
May 27, 2019
b028dff
Merge branch 'develop' into wip/1202-upload-non-images
May 29, 2019
9a63d4e
Merge branch 'develop' into wip/1202-upload-non-images
Jun 6, 2019
5784f76
Merge branch 'develop' into wip/1202-upload-non-images
Jun 6, 2019
d5aa88f
Merge branch 'develop' into wip/1202-upload-non-images
Jun 11, 2019
f53ddb0
Merge branch 'develop' into wip/1202-upload-non-images
Jun 14, 2019
44d3edb
Merge branch 'develop' into wip/1202-upload-non-images
Jun 14, 2019
80f532e
Merge branch 'develop' into wip/1202-upload-non-images
Jun 14, 2019
74dde70
Merge branch 'develop' into wip/1202-upload-non-images
Jun 20, 2019
fa7fe5f
Merge branch 'develop' into wip/1202-upload-non-images
Jun 24, 2019
6fdf1ed
Merge branch 'develop' into wip/1202-upload-non-images
Jun 24, 2019
2a2f0b6
Merge branch 'develop' into wip/1202-upload-non-images
Jul 2, 2019
36d26be
Merge branch 'develop' into wip/1202-upload-non-images
Jul 3, 2019
e63d33c
Merge branch 'develop' into wip/1202-upload-non-images
Jul 9, 2019
30c52e0
Merge branch 'develop' into wip/1202-upload-non-images
Jul 9, 2019
09b8bf8
Merge branch 'develop' into wip/1202-upload-non-images
Jul 19, 2019
c63b66a
Merge branch 'develop' into wip/1202-upload-non-images
Jul 19, 2019
f451b74
Merge branch 'develop' into wip/1202-upload-non-images
Jul 19, 2019
486e0c7
Merge branch 'develop' into wip/1202-upload-non-images
Aug 13, 2019
16dfcb4
Merge branch 'develop' into wip/1202-upload-non-images
Aug 13, 2019
6414fbf
Merge branch 'develop' into wip/1202-upload-non-images
Aug 19, 2019
17113d2
Merge branch 'develop' into wip/1202-upload-non-images
Aug 26, 2019
63937bf
Merge branch 'develop' into wip/1202-upload-non-images
Aug 27, 2019
ca119e1
Merge branch 'develop' into wip/1202-upload-non-images
Aug 29, 2019
c8bbd56
Merge branch 'develop' into wip/1202-upload-non-images
Aug 30, 2019
4c7e46e
Merge branch 'develop' into wip/1202-upload-non-images
Aug 30, 2019
a12c69e
Merge branch 'develop' into wip/1202-upload-non-images
Aug 30, 2019
1266341
Merge branch 'develop' into wip/1202-upload-non-images
Aug 30, 2019
a7bc49f
test: Fix test data.
Sep 3, 2019
ae389c7
Merge branch 'develop' into wip/1202-upload-non-images
Sep 9, 2019
25a7cc6
Merge branch 'develop' into wip/1202-upload-non-images
Sep 10, 2019
dc81e1f
Merge branch 'develop' into wip/1202-upload-non-images
Sep 12, 2019
4f803af
feat(api-v2): Support non-image files (ongoing).
Sep 16, 2019
bdb05bf
Merge branch 'develop' into wip/1202-upload-non-images
Sep 24, 2019
893773b
Merge branch 'develop' into wip/1202-upload-non-images
Sep 25, 2019
4fcc248
Merge branch 'develop' into wip/1202-upload-non-images
Oct 8, 2019
0eeb961
Merge branch 'develop' into wip/1202-upload-non-images
Oct 18, 2019
1d9a212
fix: Fix previous merge commit.
Oct 18, 2019
e054e73
Merge branch 'develop' into wip/1202-upload-non-images
Oct 18, 2019
8c6bccf
Merge branch 'develop' into wip/1202-upload-non-images
Oct 21, 2019
ecf2805
feat(api-v2): Allow upload of PDF files (ongoing).
Oct 21, 2019
6570517
test(sipi): Fix compile error.
Oct 21, 2019
b29d978
Merge branch 'develop' into wip/1202-upload-non-images
Oct 21, 2019
5d2676f
fix(sipi): Fix v1 Lua scripts.
Oct 22, 2019
80ab4df
Merge branch 'develop' into wip/1202-upload-non-images
Oct 22, 2019
6b16f50
test(sipi): Test creating a resource with a CSV file.
Oct 22, 2019
4fbb72f
Merge branch 'develop' into wip/1202-upload-non-images
Oct 23, 2019
b6d7bea
refactor(sipi): Make 500 error handling consistent in Lua scripts.
Oct 23, 2019
0752140
docs(api-v2): Document storing non-image files.
Oct 24, 2019
cd00c4c
Merge branch 'develop' into wip/1202-upload-non-images
Nov 4, 2019
9be03a7
Merge branch 'develop' into wip/1202-upload-non-images
Nov 5, 2019
7bbc80a
Merge branch 'develop' into wip/1202-upload-non-images
Nov 8, 2019
d25509b
test(sipi): Add more integration tests.
Nov 14, 2019
1f3faf4
fix(sipi): Fix docker-compose.yml.
Nov 14, 2019
a507bdd
fix(sipi): Fix typo.
Nov 14, 2019
05dbcbe
fix(sipi): Fix typo.
Nov 14, 2019
6380562
fix(sipi): Remove redundant line.
Nov 14, 2019
e712796
Merge branch 'develop' into wip/1202-upload-non-images
Nov 14, 2019
adba4b1
Merge branch 'develop' into wip/1202-upload-non-images
Nov 15, 2019
7b098cf
Merge branch 'develop' into wip/1202-upload-non-images
Nov 15, 2019
5847ef8
Merge branch 'develop' into wip/1202-upload-non-images
Nov 19, 2019
55b9557
Merge branch 'develop' into wip/1202-upload-non-images
Nov 19, 2019
d2174b1
Merge branch 'develop' into wip/1202-upload-non-images
Nov 19, 2019
746d85e
Merge branch 'develop' into wip/1202-upload-non-images
Nov 19, 2019
516e4e4
test(sipi): Check file extension on image upload.
Nov 20, 2019
60f8fce
fix(build): Increase timeout.
Nov 20, 2019
a50fec1
Merge branch 'develop' into wip/1202-upload-non-images
Nov 21, 2019
544c642
test: Fix Sipi test route.
Nov 21, 2019
5c43c9f
test: Remove println.
Nov 21, 2019
4d3688b
test: Increase GraphDB timeout.
Nov 21, 2019
3ef4e9d
Merge branch 'develop' into wip/1202-upload-non-images
Nov 26, 2019
cf17688
test: Increase Redis connection timeout.
Nov 27, 2019
5dc6e70
test: Increase timeout.
Nov 27, 2019
14abdeb
test: Revert commit to wrong branch.
Nov 27, 2019
49b2d4f
Merge branch 'develop' into wip/1202-upload-non-images
Nov 27, 2019
b4acacf
fix(api-v2): Clarify error message.
Nov 27, 2019
7e98f1e
fix(sipi): Restore code that was commented out.
Nov 28, 2019
a1a37cc
fix(sipi): Try to fix MIME type consistency check.
Nov 28, 2019
86939b8
fix(sipi): Comment out Sipi function call that doesn't work.
Nov 28, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions sipi/scripts/convert_from_path.lua
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,20 @@ if media_type == IMAGE then
return
end

local check
success, check = fullImg:mimetype_consistency(originalMimeType, originalFilename)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe you need to add

ocal submitted_mimetype
success, submitted_mimetype = server.parse_mimetype(originalMimeType)

local check
success, check = fullImg:mimetype_consistency(submitted_mimetype.mimetype, originalFilename)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added that in a1a37cc, but Sipi returns the same HTTP 500 error. I actually think that since this is an API v1 or Sipi bug, and this PR only adds functionality to API v2, there's no reason to hold back this PR because of it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok , and you are going to change the v1 behaviour anyway. If your fix doesn't solve the problem, could you open an issue in Sipi?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In other words, this problem was there before, and this PR doesn't affect it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand, but the code shouldn't have been just commented out in https://github.com/dasch-swiss/knora-api/pull/1459/files#diff-4a1a2aedd2d8b7dee2f824e33b243ca1R123

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's ok for me to make separate issues (Knora and/or Sipi) and move on with this PR

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps, but that's a different PR. You can't expect every PR to fix every bug that already existed before the PR. You're reviewing this PR.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps, but that's a different PR. You can't expect every PR to fix every bug that already existed before the PR. You're reviewing this PR.

I agree. But this is kind of a critical issue (Sipi is expected to recreate the image file in its original format), this is why I kept insisting. If it had been a detail, it would not have mattered that much.


if not success then
send_error(500, "fullImg:mimetype_consistency() failed: " .. check)
return
end

-- if check returns false, the user's input is invalid
if not check then
send_error(400, MIMETYPES_INCONSISTENCY)
return
end

local fullDims
success, fullDims = fullImg:dims()
if not success then
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -103,12 +103,12 @@ class KnoraSipiIntegrationV2ITSpec extends ITKnoraLiveSpec(KnoraSipiIntegrationV
*/
case class SipiUploadResponse(uploadedFiles: Seq[SipiUploadResponseEntry])

object GetImageMetadataResponseV2JsonProtocol extends SprayJsonSupport with DefaultJsonProtocol {
object SipiUploadResponseJsonProtocol extends SprayJsonSupport with DefaultJsonProtocol {
implicit val sipiUploadResponseEntryFormat: RootJsonFormat[SipiUploadResponseEntry] = jsonFormat4(SipiUploadResponseEntry)
implicit val sipiUploadResponseFormat: RootJsonFormat[SipiUploadResponse] = jsonFormat1(SipiUploadResponse)
}

import GetImageMetadataResponseV2JsonProtocol._
import SipiUploadResponseJsonProtocol._

/**
* Represents the information that Knora returns about an image file value that was created.
Expand Down