Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [.....XXXXX....X.............X.X.............................] duration:
3.726036745s trace:
https://jaeger.dev.storj.io/trace/273a3254470045aa
FAIL (2026-05-14 14:10:58 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.72606043s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.000583807s target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/d4412028e1c1a58
FAIL (2026-05-14 14:10:55 UTC)
orbiter warning: Get "https://version.storj.io/": dial tcp 35.209.24.65:443: i/o timeout
Duration: 30.000598595s
core
website
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [......XXXXX....X.............X.X............................] duration:
3.880144471s trace:
https://jaeger.dev.storj.io/trace/273a3254470045aa
FAIL (2026-05-14 14:10:26 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.880189477s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..........XX............X...................................] duration:
6.699175003s trace:
https://jaeger.dev.storj.io/trace/73b73ed5bcf2ff43
FAIL (2026-05-14 14:10:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.699261428s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...........XX............X..................................] duration:
14.076899382s trace:
https://jaeger.dev.storj.io/trace/73b73ed5bcf2ff43
FAIL (2026-05-14 14:10:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.076917786s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X......X..............................................] duration:
6.382814925s trace:
https://jaeger.dev.storj.io/trace/6bcf8e4c5b0862a8
FAIL (2026-05-14 14:10:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
May 14 13:36:28.324422154 (330h48m38.845245899s) [upload=316975]: Starting upload
May 14 13:36:28.324487729 (330h48m38.845311474s) [upload=316975]: Done starting upload
May 14 13:36:28.324543405 (330h48m38.84536714s) [upload=316975]: Uploading segments...
May 14 13:36:28.324757794 (330h48m38.845581539s) [upload=316975]: Got next segment. Inline: false
May 14 13:36:28.324788633 (330h48m38.845612368s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 14 13:36:28.324792991 (330h48m38.845616726s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 14 13:36:28.567718597 (330h48m39.088542342s) [seg_pos={0 0}, upload=316975]: Begin upload segment...
May 14 13:36:28.567770777 (330h48m39.088594522s) [seg_pos={0 0}, upload=316975]: Done begin upload segment.
May 14 13:36:28.567794703 (330h48m39.088618438s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.567819981 (330h48m39.088643726s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568338951 (330h48m39.089162696s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568829707 (330h48m39.089653452s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.569727691 (330h48m39.090551436s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570335932 (330h48m39.091159677s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570398291 (330h48m39.091222026s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570698164 (330h48m39.091521899s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570954873 (330h48m39.091778618s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571296846 (330h48m39.092120581s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571662814 (330h48m39.092486549s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572289079 (330h48m39.093112825s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572752583 (330h48m39.093576329s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573191752 (330h48m39.094015497s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573295591 (330h48m39.094119336s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573700533 (330h48m39.094524288s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:29.885678690 (330h48m40.406502435s) [upload=316975]: Next returned nil segment
May 14 13:36:29.885693760 (330h48m40.406517505s) [upload=316975]: Waiting for error group managing segments...
May 14 13:36:30.267895418 (330h48m40.788719153s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.436072812 (330h48m40.956896557s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.463303156 (330h48m40.984126901s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.495900862 (330h48m41.016724608s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.545724080 (330h48m41.066547825s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.796939386 (330h48m41.317763121s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.820623042 (330h48m41.341446847s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.914048005 (330h48m41.43487174s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:31.417499576 (330h48m41.938323331s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:32.451371893 (330h48m42.972195648s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:37:27.674200183 (330h49m38.195023918s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12LPgzYrzGJTay9LE1ZNm3dv9vhtj4qQqK5gMdc8B9nyBXdYLuw, address:193.149.164.67:4015): protocol: rpc: dial tcp 193.149.164.67:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674447916 (330h49m38.195271701s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1xK3trpsZJrYRGZteFknm89ykujQsEnjdYqTXYbWJEK4yp8Wio, address:193.149.164.69:4070): protocol: rpc: dial tcp 193.149.164.69:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674557244 (330h49m38.195380989s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12sCyZhtEDaFcQfcCsETubryphHfHcWzYay5pYMP5xhCuzTUaSd, address:193.149.164.66:4988): protocol: rpc: dial tcp 193.149.164.66:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674850474 (330h49m38.195674219s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1Ce95mJEDhVmLMjxKY6giHtJDG9gduHXDoNmvZRpmFPAZZTmC2, address:193.149.164.71:4310): rpc: dial tcp 193.149.164.71:4310: i/o timeout
May 14 13:37:27.674997264 (330h49m38.195821009s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12aVyaTYczsdh8UPxFVHrcKjxHxz75b3Q67BcivVDLtM3beaSLm, address:193.149.164.70:4108): protocol: write tcp 188.166.225.170:41018->193.149.164.70:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.675143844 (330h49m38.195967589s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12UEesqk1Bez6iG4eUgV5Kpy1mM2zFmA3bVgCU4w5jF4KuX8DQP, address:193.149.164.68:4060): protocol: rpc: dial tcp 193.149.164.68:4060: i/o timeout; context deadline exceeded
May 14 13:37:27.675178129 (330h49m38.196001874s) [seg_pos={0 0}, upload=316975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
May 14 13:37:27.920638022 (330h49m38.441461777s) [upload=316975]: Done uploading segments...
May 14 13:37:27.920680102 (330h49m38.441503907s) [upload=316975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 6.382841786s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [.......XXXXX....X.............X.X...........................] duration:
3.942361524s trace:
https://jaeger.dev.storj.io/trace/273a3254470045aa
FAIL (2026-05-14 14:09:20 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.942387535s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [............XX............X.................................] duration:
6.591191092s trace:
https://jaeger.dev.storj.io/trace/73b73ed5bcf2ff43
FAIL (2026-05-14 14:09:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.591212373s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X......X.............................................] duration:
5.957648993s trace:
https://jaeger.dev.storj.io/trace/6bcf8e4c5b0862a8
FAIL (2026-05-14 14:09:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
May 14 13:36:28.324422154 (330h48m38.845245899s) [upload=316975]: Starting upload
May 14 13:36:28.324487729 (330h48m38.845311474s) [upload=316975]: Done starting upload
May 14 13:36:28.324543405 (330h48m38.84536714s) [upload=316975]: Uploading segments...
May 14 13:36:28.324757794 (330h48m38.845581539s) [upload=316975]: Got next segment. Inline: false
May 14 13:36:28.324788633 (330h48m38.845612368s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 14 13:36:28.324792991 (330h48m38.845616726s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 14 13:36:28.567718597 (330h48m39.088542342s) [seg_pos={0 0}, upload=316975]: Begin upload segment...
May 14 13:36:28.567770777 (330h48m39.088594522s) [seg_pos={0 0}, upload=316975]: Done begin upload segment.
May 14 13:36:28.567794703 (330h48m39.088618438s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.567819981 (330h48m39.088643726s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568338951 (330h48m39.089162696s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568829707 (330h48m39.089653452s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.569727691 (330h48m39.090551436s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570335932 (330h48m39.091159677s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570398291 (330h48m39.091222026s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570698164 (330h48m39.091521899s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570954873 (330h48m39.091778618s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571296846 (330h48m39.092120581s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571662814 (330h48m39.092486549s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572289079 (330h48m39.093112825s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572752583 (330h48m39.093576329s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573191752 (330h48m39.094015497s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573295591 (330h48m39.094119336s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573700533 (330h48m39.094524288s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:29.885678690 (330h48m40.406502435s) [upload=316975]: Next returned nil segment
May 14 13:36:29.885693760 (330h48m40.406517505s) [upload=316975]: Waiting for error group managing segments...
May 14 13:36:30.267895418 (330h48m40.788719153s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.436072812 (330h48m40.956896557s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.463303156 (330h48m40.984126901s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.495900862 (330h48m41.016724608s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.545724080 (330h48m41.066547825s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.796939386 (330h48m41.317763121s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.820623042 (330h48m41.341446847s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.914048005 (330h48m41.43487174s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:31.417499576 (330h48m41.938323331s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:32.451371893 (330h48m42.972195648s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:37:27.674200183 (330h49m38.195023918s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12LPgzYrzGJTay9LE1ZNm3dv9vhtj4qQqK5gMdc8B9nyBXdYLuw, address:193.149.164.67:4015): protocol: rpc: dial tcp 193.149.164.67:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674447916 (330h49m38.195271701s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1xK3trpsZJrYRGZteFknm89ykujQsEnjdYqTXYbWJEK4yp8Wio, address:193.149.164.69:4070): protocol: rpc: dial tcp 193.149.164.69:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674557244 (330h49m38.195380989s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12sCyZhtEDaFcQfcCsETubryphHfHcWzYay5pYMP5xhCuzTUaSd, address:193.149.164.66:4988): protocol: rpc: dial tcp 193.149.164.66:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674850474 (330h49m38.195674219s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1Ce95mJEDhVmLMjxKY6giHtJDG9gduHXDoNmvZRpmFPAZZTmC2, address:193.149.164.71:4310): rpc: dial tcp 193.149.164.71:4310: i/o timeout
May 14 13:37:27.674997264 (330h49m38.195821009s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12aVyaTYczsdh8UPxFVHrcKjxHxz75b3Q67BcivVDLtM3beaSLm, address:193.149.164.70:4108): protocol: write tcp 188.166.225.170:41018->193.149.164.70:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.675143844 (330h49m38.195967589s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12UEesqk1Bez6iG4eUgV5Kpy1mM2zFmA3bVgCU4w5jF4KuX8DQP, address:193.149.164.68:4060): protocol: rpc: dial tcp 193.149.164.68:4060: i/o timeout; context deadline exceeded
May 14 13:37:27.675178129 (330h49m38.196001874s) [seg_pos={0 0}, upload=316975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
May 14 13:37:27.920638022 (330h49m38.441461777s) [upload=316975]: Done uploading segments...
May 14 13:37:27.920680102 (330h49m38.441503907s) [upload=316975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 5.957670505s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [........XXXXX....X.............X.X..........................] duration:
3.995994879s trace:
https://jaeger.dev.storj.io/trace/273a3254470045aa
FAIL (2026-05-14 14:08:58 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.996098646s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [........X......X............................................] duration:
6.965008805s trace:
https://jaeger.dev.storj.io/trace/6bcf8e4c5b0862a8
FAIL (2026-05-14 14:08:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
May 14 13:36:28.324422154 (330h48m38.845245899s) [upload=316975]: Starting upload
May 14 13:36:28.324487729 (330h48m38.845311474s) [upload=316975]: Done starting upload
May 14 13:36:28.324543405 (330h48m38.84536714s) [upload=316975]: Uploading segments...
May 14 13:36:28.324757794 (330h48m38.845581539s) [upload=316975]: Got next segment. Inline: false
May 14 13:36:28.324788633 (330h48m38.845612368s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 14 13:36:28.324792991 (330h48m38.845616726s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 14 13:36:28.567718597 (330h48m39.088542342s) [seg_pos={0 0}, upload=316975]: Begin upload segment...
May 14 13:36:28.567770777 (330h48m39.088594522s) [seg_pos={0 0}, upload=316975]: Done begin upload segment.
May 14 13:36:28.567794703 (330h48m39.088618438s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.567819981 (330h48m39.088643726s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568338951 (330h48m39.089162696s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568829707 (330h48m39.089653452s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.569727691 (330h48m39.090551436s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570335932 (330h48m39.091159677s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570398291 (330h48m39.091222026s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570698164 (330h48m39.091521899s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570954873 (330h48m39.091778618s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571296846 (330h48m39.092120581s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571662814 (330h48m39.092486549s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572289079 (330h48m39.093112825s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572752583 (330h48m39.093576329s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573191752 (330h48m39.094015497s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573295591 (330h48m39.094119336s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573700533 (330h48m39.094524288s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:29.885678690 (330h48m40.406502435s) [upload=316975]: Next returned nil segment
May 14 13:36:29.885693760 (330h48m40.406517505s) [upload=316975]: Waiting for error group managing segments...
May 14 13:36:30.267895418 (330h48m40.788719153s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.436072812 (330h48m40.956896557s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.463303156 (330h48m40.984126901s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.495900862 (330h48m41.016724608s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.545724080 (330h48m41.066547825s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.796939386 (330h48m41.317763121s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.820623042 (330h48m41.341446847s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.914048005 (330h48m41.43487174s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:31.417499576 (330h48m41.938323331s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:32.451371893 (330h48m42.972195648s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:37:27.674200183 (330h49m38.195023918s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12LPgzYrzGJTay9LE1ZNm3dv9vhtj4qQqK5gMdc8B9nyBXdYLuw, address:193.149.164.67:4015): protocol: rpc: dial tcp 193.149.164.67:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674447916 (330h49m38.195271701s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1xK3trpsZJrYRGZteFknm89ykujQsEnjdYqTXYbWJEK4yp8Wio, address:193.149.164.69:4070): protocol: rpc: dial tcp 193.149.164.69:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674557244 (330h49m38.195380989s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12sCyZhtEDaFcQfcCsETubryphHfHcWzYay5pYMP5xhCuzTUaSd, address:193.149.164.66:4988): protocol: rpc: dial tcp 193.149.164.66:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674850474 (330h49m38.195674219s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1Ce95mJEDhVmLMjxKY6giHtJDG9gduHXDoNmvZRpmFPAZZTmC2, address:193.149.164.71:4310): rpc: dial tcp 193.149.164.71:4310: i/o timeout
May 14 13:37:27.674997264 (330h49m38.195821009s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12aVyaTYczsdh8UPxFVHrcKjxHxz75b3Q67BcivVDLtM3beaSLm, address:193.149.164.70:4108): protocol: write tcp 188.166.225.170:41018->193.149.164.70:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.675143844 (330h49m38.195967589s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12UEesqk1Bez6iG4eUgV5Kpy1mM2zFmA3bVgCU4w5jF4KuX8DQP, address:193.149.164.68:4060): protocol: rpc: dial tcp 193.149.164.68:4060: i/o timeout; context deadline exceeded
May 14 13:37:27.675178129 (330h49m38.196001874s) [seg_pos={0 0}, upload=316975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
May 14 13:37:27.920638022 (330h49m38.441461777s) [upload=316975]: Done uploading segments...
May 14 13:37:27.920680102 (330h49m38.441503907s) [upload=316975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 6.965035625s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.............XX............X................................] duration:
6.789566672s trace:
https://jaeger.dev.storj.io/trace/73b73ed5bcf2ff43
FAIL (2026-05-14 14:08:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.789588233s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [.........XXXXX....X.............X.X.........................] duration:
3.957543299s trace:
https://jaeger.dev.storj.io/trace/273a3254470045aa
FAIL (2026-05-14 14:08:06 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.957567274s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..............XX............X...............................] duration:
6.704932086s trace:
https://jaeger.dev.storj.io/trace/73b73ed5bcf2ff43
FAIL (2026-05-14 14:08:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.704997251s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........X......X...........................................] duration:
6.493910775s trace:
https://jaeger.dev.storj.io/trace/6bcf8e4c5b0862a8
FAIL (2026-05-14 14:07:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
May 14 13:36:28.324422154 (330h48m38.845245899s) [upload=316975]: Starting upload
May 14 13:36:28.324487729 (330h48m38.845311474s) [upload=316975]: Done starting upload
May 14 13:36:28.324543405 (330h48m38.84536714s) [upload=316975]: Uploading segments...
May 14 13:36:28.324757794 (330h48m38.845581539s) [upload=316975]: Got next segment. Inline: false
May 14 13:36:28.324788633 (330h48m38.845612368s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 14 13:36:28.324792991 (330h48m38.845616726s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 14 13:36:28.567718597 (330h48m39.088542342s) [seg_pos={0 0}, upload=316975]: Begin upload segment...
May 14 13:36:28.567770777 (330h48m39.088594522s) [seg_pos={0 0}, upload=316975]: Done begin upload segment.
May 14 13:36:28.567794703 (330h48m39.088618438s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.567819981 (330h48m39.088643726s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568338951 (330h48m39.089162696s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568829707 (330h48m39.089653452s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.569727691 (330h48m39.090551436s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570335932 (330h48m39.091159677s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570398291 (330h48m39.091222026s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570698164 (330h48m39.091521899s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570954873 (330h48m39.091778618s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571296846 (330h48m39.092120581s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571662814 (330h48m39.092486549s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572289079 (330h48m39.093112825s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572752583 (330h48m39.093576329s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573191752 (330h48m39.094015497s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573295591 (330h48m39.094119336s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573700533 (330h48m39.094524288s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:29.885678690 (330h48m40.406502435s) [upload=316975]: Next returned nil segment
May 14 13:36:29.885693760 (330h48m40.406517505s) [upload=316975]: Waiting for error group managing segments...
May 14 13:36:30.267895418 (330h48m40.788719153s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.436072812 (330h48m40.956896557s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.463303156 (330h48m40.984126901s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.495900862 (330h48m41.016724608s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.545724080 (330h48m41.066547825s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.796939386 (330h48m41.317763121s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.820623042 (330h48m41.341446847s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.914048005 (330h48m41.43487174s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:31.417499576 (330h48m41.938323331s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:32.451371893 (330h48m42.972195648s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:37:27.674200183 (330h49m38.195023918s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12LPgzYrzGJTay9LE1ZNm3dv9vhtj4qQqK5gMdc8B9nyBXdYLuw, address:193.149.164.67:4015): protocol: rpc: dial tcp 193.149.164.67:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674447916 (330h49m38.195271701s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1xK3trpsZJrYRGZteFknm89ykujQsEnjdYqTXYbWJEK4yp8Wio, address:193.149.164.69:4070): protocol: rpc: dial tcp 193.149.164.69:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674557244 (330h49m38.195380989s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12sCyZhtEDaFcQfcCsETubryphHfHcWzYay5pYMP5xhCuzTUaSd, address:193.149.164.66:4988): protocol: rpc: dial tcp 193.149.164.66:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674850474 (330h49m38.195674219s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1Ce95mJEDhVmLMjxKY6giHtJDG9gduHXDoNmvZRpmFPAZZTmC2, address:193.149.164.71:4310): rpc: dial tcp 193.149.164.71:4310: i/o timeout
May 14 13:37:27.674997264 (330h49m38.195821009s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12aVyaTYczsdh8UPxFVHrcKjxHxz75b3Q67BcivVDLtM3beaSLm, address:193.149.164.70:4108): protocol: write tcp 188.166.225.170:41018->193.149.164.70:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.675143844 (330h49m38.195967589s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12UEesqk1Bez6iG4eUgV5Kpy1mM2zFmA3bVgCU4w5jF4KuX8DQP, address:193.149.164.68:4060): protocol: rpc: dial tcp 193.149.164.68:4060: i/o timeout; context deadline exceeded
May 14 13:37:27.675178129 (330h49m38.196001874s) [seg_pos={0 0}, upload=316975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
May 14 13:37:27.920638022 (330h49m38.441461777s) [upload=316975]: Done uploading segments...
May 14 13:37:27.920680102 (330h49m38.441503907s) [upload=316975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 6.493938548s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [..........XXXXX....X.............X.X........................] duration:
3.880852814s trace:
https://jaeger.dev.storj.io/trace/273a3254470045aa
FAIL (2026-05-14 14:07:17 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.880876809s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........X......X..........................................] duration:
6.371416992s trace:
https://jaeger.dev.storj.io/trace/6bcf8e4c5b0862a8
FAIL (2026-05-14 14:07:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
May 14 13:36:28.324422154 (330h48m38.845245899s) [upload=316975]: Starting upload
May 14 13:36:28.324487729 (330h48m38.845311474s) [upload=316975]: Done starting upload
May 14 13:36:28.324543405 (330h48m38.84536714s) [upload=316975]: Uploading segments...
May 14 13:36:28.324757794 (330h48m38.845581539s) [upload=316975]: Got next segment. Inline: false
May 14 13:36:28.324788633 (330h48m38.845612368s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 14 13:36:28.324792991 (330h48m38.845616726s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 14 13:36:28.567718597 (330h48m39.088542342s) [seg_pos={0 0}, upload=316975]: Begin upload segment...
May 14 13:36:28.567770777 (330h48m39.088594522s) [seg_pos={0 0}, upload=316975]: Done begin upload segment.
May 14 13:36:28.567794703 (330h48m39.088618438s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.567819981 (330h48m39.088643726s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568338951 (330h48m39.089162696s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568829707 (330h48m39.089653452s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.569727691 (330h48m39.090551436s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570335932 (330h48m39.091159677s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570398291 (330h48m39.091222026s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570698164 (330h48m39.091521899s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570954873 (330h48m39.091778618s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571296846 (330h48m39.092120581s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571662814 (330h48m39.092486549s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572289079 (330h48m39.093112825s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572752583 (330h48m39.093576329s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573191752 (330h48m39.094015497s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573295591 (330h48m39.094119336s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573700533 (330h48m39.094524288s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:29.885678690 (330h48m40.406502435s) [upload=316975]: Next returned nil segment
May 14 13:36:29.885693760 (330h48m40.406517505s) [upload=316975]: Waiting for error group managing segments...
May 14 13:36:30.267895418 (330h48m40.788719153s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.436072812 (330h48m40.956896557s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.463303156 (330h48m40.984126901s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.495900862 (330h48m41.016724608s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.545724080 (330h48m41.066547825s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.796939386 (330h48m41.317763121s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.820623042 (330h48m41.341446847s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.914048005 (330h48m41.43487174s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:31.417499576 (330h48m41.938323331s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:32.451371893 (330h48m42.972195648s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:37:27.674200183 (330h49m38.195023918s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12LPgzYrzGJTay9LE1ZNm3dv9vhtj4qQqK5gMdc8B9nyBXdYLuw, address:193.149.164.67:4015): protocol: rpc: dial tcp 193.149.164.67:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674447916 (330h49m38.195271701s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1xK3trpsZJrYRGZteFknm89ykujQsEnjdYqTXYbWJEK4yp8Wio, address:193.149.164.69:4070): protocol: rpc: dial tcp 193.149.164.69:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674557244 (330h49m38.195380989s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12sCyZhtEDaFcQfcCsETubryphHfHcWzYay5pYMP5xhCuzTUaSd, address:193.149.164.66:4988): protocol: rpc: dial tcp 193.149.164.66:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674850474 (330h49m38.195674219s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1Ce95mJEDhVmLMjxKY6giHtJDG9gduHXDoNmvZRpmFPAZZTmC2, address:193.149.164.71:4310): rpc: dial tcp 193.149.164.71:4310: i/o timeout
May 14 13:37:27.674997264 (330h49m38.195821009s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12aVyaTYczsdh8UPxFVHrcKjxHxz75b3Q67BcivVDLtM3beaSLm, address:193.149.164.70:4108): protocol: write tcp 188.166.225.170:41018->193.149.164.70:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.675143844 (330h49m38.195967589s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12UEesqk1Bez6iG4eUgV5Kpy1mM2zFmA3bVgCU4w5jF4KuX8DQP, address:193.149.164.68:4060): protocol: rpc: dial tcp 193.149.164.68:4060: i/o timeout; context deadline exceeded
May 14 13:37:27.675178129 (330h49m38.196001874s) [seg_pos={0 0}, upload=316975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
May 14 13:37:27.920638022 (330h49m38.441461777s) [upload=316975]: Done uploading segments...
May 14 13:37:27.920680102 (330h49m38.441503907s) [upload=316975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 6.371433323s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...............XX............X..............................] duration:
6.667443721s trace:
https://jaeger.dev.storj.io/trace/73b73ed5bcf2ff43
FAIL (2026-05-14 14:07:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.667471665s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........X......X.........................................] duration:
6.656577775s trace:
https://jaeger.dev.storj.io/trace/6bcf8e4c5b0862a8
FAIL (2026-05-14 14:06:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
May 14 13:36:28.324422154 (330h48m38.845245899s) [upload=316975]: Starting upload
May 14 13:36:28.324487729 (330h48m38.845311474s) [upload=316975]: Done starting upload
May 14 13:36:28.324543405 (330h48m38.84536714s) [upload=316975]: Uploading segments...
May 14 13:36:28.324757794 (330h48m38.845581539s) [upload=316975]: Got next segment. Inline: false
May 14 13:36:28.324788633 (330h48m38.845612368s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 14 13:36:28.324792991 (330h48m38.845616726s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 14 13:36:28.567718597 (330h48m39.088542342s) [seg_pos={0 0}, upload=316975]: Begin upload segment...
May 14 13:36:28.567770777 (330h48m39.088594522s) [seg_pos={0 0}, upload=316975]: Done begin upload segment.
May 14 13:36:28.567794703 (330h48m39.088618438s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.567819981 (330h48m39.088643726s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568338951 (330h48m39.089162696s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568829707 (330h48m39.089653452s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.569727691 (330h48m39.090551436s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570335932 (330h48m39.091159677s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570398291 (330h48m39.091222026s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570698164 (330h48m39.091521899s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570954873 (330h48m39.091778618s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571296846 (330h48m39.092120581s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571662814 (330h48m39.092486549s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572289079 (330h48m39.093112825s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572752583 (330h48m39.093576329s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573191752 (330h48m39.094015497s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573295591 (330h48m39.094119336s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573700533 (330h48m39.094524288s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:29.885678690 (330h48m40.406502435s) [upload=316975]: Next returned nil segment
May 14 13:36:29.885693760 (330h48m40.406517505s) [upload=316975]: Waiting for error group managing segments...
May 14 13:36:30.267895418 (330h48m40.788719153s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.436072812 (330h48m40.956896557s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.463303156 (330h48m40.984126901s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.495900862 (330h48m41.016724608s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.545724080 (330h48m41.066547825s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.796939386 (330h48m41.317763121s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.820623042 (330h48m41.341446847s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.914048005 (330h48m41.43487174s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:31.417499576 (330h48m41.938323331s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:32.451371893 (330h48m42.972195648s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:37:27.674200183 (330h49m38.195023918s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12LPgzYrzGJTay9LE1ZNm3dv9vhtj4qQqK5gMdc8B9nyBXdYLuw, address:193.149.164.67:4015): protocol: rpc: dial tcp 193.149.164.67:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674447916 (330h49m38.195271701s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1xK3trpsZJrYRGZteFknm89ykujQsEnjdYqTXYbWJEK4yp8Wio, address:193.149.164.69:4070): protocol: rpc: dial tcp 193.149.164.69:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674557244 (330h49m38.195380989s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12sCyZhtEDaFcQfcCsETubryphHfHcWzYay5pYMP5xhCuzTUaSd, address:193.149.164.66:4988): protocol: rpc: dial tcp 193.149.164.66:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674850474 (330h49m38.195674219s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1Ce95mJEDhVmLMjxKY6giHtJDG9gduHXDoNmvZRpmFPAZZTmC2, address:193.149.164.71:4310): rpc: dial tcp 193.149.164.71:4310: i/o timeout
May 14 13:37:27.674997264 (330h49m38.195821009s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12aVyaTYczsdh8UPxFVHrcKjxHxz75b3Q67BcivVDLtM3beaSLm, address:193.149.164.70:4108): protocol: write tcp 188.166.225.170:41018->193.149.164.70:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.675143844 (330h49m38.195967589s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12UEesqk1Bez6iG4eUgV5Kpy1mM2zFmA3bVgCU4w5jF4KuX8DQP, address:193.149.164.68:4060): protocol: rpc: dial tcp 193.149.164.68:4060: i/o timeout; context deadline exceeded
May 14 13:37:27.675178129 (330h49m38.196001874s) [seg_pos={0 0}, upload=316975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
May 14 13:37:27.920638022 (330h49m38.441461777s) [upload=316975]: Done uploading segments...
May 14 13:37:27.920680102 (330h49m38.441503907s) [upload=316975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 6.656602351s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [................XX............X.............................] duration:
6.737897173s trace:
https://jaeger.dev.storj.io/trace/73b73ed5bcf2ff43
FAIL (2026-05-14 14:06:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.73792261s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [...........XXXXX....X.............X.X.......................] duration:
3.766787703s trace:
https://jaeger.dev.storj.io/trace/273a3254470045aa
FAIL (2026-05-14 14:06:22 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.766809856s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [............X......X........................................] duration:
7.101853171s trace:
https://jaeger.dev.storj.io/trace/6bcf8e4c5b0862a8
FAIL (2026-05-14 14:06:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
May 14 13:36:28.324422154 (330h48m38.845245899s) [upload=316975]: Starting upload
May 14 13:36:28.324487729 (330h48m38.845311474s) [upload=316975]: Done starting upload
May 14 13:36:28.324543405 (330h48m38.84536714s) [upload=316975]: Uploading segments...
May 14 13:36:28.324757794 (330h48m38.845581539s) [upload=316975]: Got next segment. Inline: false
May 14 13:36:28.324788633 (330h48m38.845612368s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 14 13:36:28.324792991 (330h48m38.845616726s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 14 13:36:28.567718597 (330h48m39.088542342s) [seg_pos={0 0}, upload=316975]: Begin upload segment...
May 14 13:36:28.567770777 (330h48m39.088594522s) [seg_pos={0 0}, upload=316975]: Done begin upload segment.
May 14 13:36:28.567794703 (330h48m39.088618438s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.567819981 (330h48m39.088643726s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568338951 (330h48m39.089162696s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568829707 (330h48m39.089653452s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.569727691 (330h48m39.090551436s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570335932 (330h48m39.091159677s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570398291 (330h48m39.091222026s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570698164 (330h48m39.091521899s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570954873 (330h48m39.091778618s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571296846 (330h48m39.092120581s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571662814 (330h48m39.092486549s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572289079 (330h48m39.093112825s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572752583 (330h48m39.093576329s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573191752 (330h48m39.094015497s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573295591 (330h48m39.094119336s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573700533 (330h48m39.094524288s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:29.885678690 (330h48m40.406502435s) [upload=316975]: Next returned nil segment
May 14 13:36:29.885693760 (330h48m40.406517505s) [upload=316975]: Waiting for error group managing segments...
May 14 13:36:30.267895418 (330h48m40.788719153s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.436072812 (330h48m40.956896557s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.463303156 (330h48m40.984126901s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.495900862 (330h48m41.016724608s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.545724080 (330h48m41.066547825s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.796939386 (330h48m41.317763121s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.820623042 (330h48m41.341446847s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.914048005 (330h48m41.43487174s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:31.417499576 (330h48m41.938323331s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:32.451371893 (330h48m42.972195648s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:37:27.674200183 (330h49m38.195023918s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12LPgzYrzGJTay9LE1ZNm3dv9vhtj4qQqK5gMdc8B9nyBXdYLuw, address:193.149.164.67:4015): protocol: rpc: dial tcp 193.149.164.67:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674447916 (330h49m38.195271701s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1xK3trpsZJrYRGZteFknm89ykujQsEnjdYqTXYbWJEK4yp8Wio, address:193.149.164.69:4070): protocol: rpc: dial tcp 193.149.164.69:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674557244 (330h49m38.195380989s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12sCyZhtEDaFcQfcCsETubryphHfHcWzYay5pYMP5xhCuzTUaSd, address:193.149.164.66:4988): protocol: rpc: dial tcp 193.149.164.66:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674850474 (330h49m38.195674219s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1Ce95mJEDhVmLMjxKY6giHtJDG9gduHXDoNmvZRpmFPAZZTmC2, address:193.149.164.71:4310): rpc: dial tcp 193.149.164.71:4310: i/o timeout
May 14 13:37:27.674997264 (330h49m38.195821009s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12aVyaTYczsdh8UPxFVHrcKjxHxz75b3Q67BcivVDLtM3beaSLm, address:193.149.164.70:4108): protocol: write tcp 188.166.225.170:41018->193.149.164.70:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.675143844 (330h49m38.195967589s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12UEesqk1Bez6iG4eUgV5Kpy1mM2zFmA3bVgCU4w5jF4KuX8DQP, address:193.149.164.68:4060): protocol: rpc: dial tcp 193.149.164.68:4060: i/o timeout; context deadline exceeded
May 14 13:37:27.675178129 (330h49m38.196001874s) [seg_pos={0 0}, upload=316975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
May 14 13:37:27.920638022 (330h49m38.441461777s) [upload=316975]: Done uploading segments...
May 14 13:37:27.920680102 (330h49m38.441503907s) [upload=316975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 7.101871485s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.................XX............X............................] duration:
13.330316385s trace:
https://jaeger.dev.storj.io/trace/73b73ed5bcf2ff43
FAIL (2026-05-14 14:05:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 13.330330892s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............X......X.......................................] duration:
6.3967895s trace:
https://jaeger.dev.storj.io/trace/6bcf8e4c5b0862a8
FAIL (2026-05-14 14:05:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
May 14 13:36:28.324422154 (330h48m38.845245899s) [upload=316975]: Starting upload
May 14 13:36:28.324487729 (330h48m38.845311474s) [upload=316975]: Done starting upload
May 14 13:36:28.324543405 (330h48m38.84536714s) [upload=316975]: Uploading segments...
May 14 13:36:28.324757794 (330h48m38.845581539s) [upload=316975]: Got next segment. Inline: false
May 14 13:36:28.324788633 (330h48m38.845612368s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 14 13:36:28.324792991 (330h48m38.845616726s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 14 13:36:28.567718597 (330h48m39.088542342s) [seg_pos={0 0}, upload=316975]: Begin upload segment...
May 14 13:36:28.567770777 (330h48m39.088594522s) [seg_pos={0 0}, upload=316975]: Done begin upload segment.
May 14 13:36:28.567794703 (330h48m39.088618438s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.567819981 (330h48m39.088643726s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568338951 (330h48m39.089162696s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568829707 (330h48m39.089653452s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.569727691 (330h48m39.090551436s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570335932 (330h48m39.091159677s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570398291 (330h48m39.091222026s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570698164 (330h48m39.091521899s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570954873 (330h48m39.091778618s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571296846 (330h48m39.092120581s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571662814 (330h48m39.092486549s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572289079 (330h48m39.093112825s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572752583 (330h48m39.093576329s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573191752 (330h48m39.094015497s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573295591 (330h48m39.094119336s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573700533 (330h48m39.094524288s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:29.885678690 (330h48m40.406502435s) [upload=316975]: Next returned nil segment
May 14 13:36:29.885693760 (330h48m40.406517505s) [upload=316975]: Waiting for error group managing segments...
May 14 13:36:30.267895418 (330h48m40.788719153s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.436072812 (330h48m40.956896557s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.463303156 (330h48m40.984126901s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.495900862 (330h48m41.016724608s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.545724080 (330h48m41.066547825s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.796939386 (330h48m41.317763121s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.820623042 (330h48m41.341446847s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.914048005 (330h48m41.43487174s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:31.417499576 (330h48m41.938323331s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:32.451371893 (330h48m42.972195648s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:37:27.674200183 (330h49m38.195023918s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12LPgzYrzGJTay9LE1ZNm3dv9vhtj4qQqK5gMdc8B9nyBXdYLuw, address:193.149.164.67:4015): protocol: rpc: dial tcp 193.149.164.67:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674447916 (330h49m38.195271701s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1xK3trpsZJrYRGZteFknm89ykujQsEnjdYqTXYbWJEK4yp8Wio, address:193.149.164.69:4070): protocol: rpc: dial tcp 193.149.164.69:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674557244 (330h49m38.195380989s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12sCyZhtEDaFcQfcCsETubryphHfHcWzYay5pYMP5xhCuzTUaSd, address:193.149.164.66:4988): protocol: rpc: dial tcp 193.149.164.66:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674850474 (330h49m38.195674219s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1Ce95mJEDhVmLMjxKY6giHtJDG9gduHXDoNmvZRpmFPAZZTmC2, address:193.149.164.71:4310): rpc: dial tcp 193.149.164.71:4310: i/o timeout
May 14 13:37:27.674997264 (330h49m38.195821009s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12aVyaTYczsdh8UPxFVHrcKjxHxz75b3Q67BcivVDLtM3beaSLm, address:193.149.164.70:4108): protocol: write tcp 188.166.225.170:41018->193.149.164.70:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.675143844 (330h49m38.195967589s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12UEesqk1Bez6iG4eUgV5Kpy1mM2zFmA3bVgCU4w5jF4KuX8DQP, address:193.149.164.68:4060): protocol: rpc: dial tcp 193.149.164.68:4060: i/o timeout; context deadline exceeded
May 14 13:37:27.675178129 (330h49m38.196001874s) [seg_pos={0 0}, upload=316975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
May 14 13:37:27.920638022 (330h49m38.441461777s) [upload=316975]: Done uploading segments...
May 14 13:37:27.920680102 (330h49m38.441503907s) [upload=316975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 6.396818416s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [............XXXXX....X.............X.X......................] duration:
3.755285795s trace:
https://jaeger.dev.storj.io/trace/273a3254470045aa
FAIL (2026-05-14 14:05:07 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.755308618s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..................XX............X...........................] duration:
6.644022899s trace:
https://jaeger.dev.storj.io/trace/73b73ed5bcf2ff43
FAIL (2026-05-14 14:05:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.644044622s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [.............XXXXX....X.............X.X.....................] duration:
3.894267924s trace:
https://jaeger.dev.storj.io/trace/273a3254470045aa
FAIL (2026-05-14 14:04:32 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.894285678s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............X......X......................................] duration:
8.478715592s trace:
https://jaeger.dev.storj.io/trace/6bcf8e4c5b0862a8
FAIL (2026-05-14 14:03:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
May 14 13:36:28.324422154 (330h48m38.845245899s) [upload=316975]: Starting upload
May 14 13:36:28.324487729 (330h48m38.845311474s) [upload=316975]: Done starting upload
May 14 13:36:28.324543405 (330h48m38.84536714s) [upload=316975]: Uploading segments...
May 14 13:36:28.324757794 (330h48m38.845581539s) [upload=316975]: Got next segment. Inline: false
May 14 13:36:28.324788633 (330h48m38.845612368s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 14 13:36:28.324792991 (330h48m38.845616726s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 14 13:36:28.567718597 (330h48m39.088542342s) [seg_pos={0 0}, upload=316975]: Begin upload segment...
May 14 13:36:28.567770777 (330h48m39.088594522s) [seg_pos={0 0}, upload=316975]: Done begin upload segment.
May 14 13:36:28.567794703 (330h48m39.088618438s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.567819981 (330h48m39.088643726s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568338951 (330h48m39.089162696s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568829707 (330h48m39.089653452s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.569727691 (330h48m39.090551436s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570335932 (330h48m39.091159677s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570398291 (330h48m39.091222026s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570698164 (330h48m39.091521899s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570954873 (330h48m39.091778618s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571296846 (330h48m39.092120581s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571662814 (330h48m39.092486549s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572289079 (330h48m39.093112825s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572752583 (330h48m39.093576329s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573191752 (330h48m39.094015497s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573295591 (330h48m39.094119336s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573700533 (330h48m39.094524288s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:29.885678690 (330h48m40.406502435s) [upload=316975]: Next returned nil segment
May 14 13:36:29.885693760 (330h48m40.406517505s) [upload=316975]: Waiting for error group managing segments...
May 14 13:36:30.267895418 (330h48m40.788719153s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.436072812 (330h48m40.956896557s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.463303156 (330h48m40.984126901s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.495900862 (330h48m41.016724608s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.545724080 (330h48m41.066547825s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.796939386 (330h48m41.317763121s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.820623042 (330h48m41.341446847s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.914048005 (330h48m41.43487174s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:31.417499576 (330h48m41.938323331s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:32.451371893 (330h48m42.972195648s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:37:27.674200183 (330h49m38.195023918s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12LPgzYrzGJTay9LE1ZNm3dv9vhtj4qQqK5gMdc8B9nyBXdYLuw, address:193.149.164.67:4015): protocol: rpc: dial tcp 193.149.164.67:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674447916 (330h49m38.195271701s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1xK3trpsZJrYRGZteFknm89ykujQsEnjdYqTXYbWJEK4yp8Wio, address:193.149.164.69:4070): protocol: rpc: dial tcp 193.149.164.69:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674557244 (330h49m38.195380989s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12sCyZhtEDaFcQfcCsETubryphHfHcWzYay5pYMP5xhCuzTUaSd, address:193.149.164.66:4988): protocol: rpc: dial tcp 193.149.164.66:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674850474 (330h49m38.195674219s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1Ce95mJEDhVmLMjxKY6giHtJDG9gduHXDoNmvZRpmFPAZZTmC2, address:193.149.164.71:4310): rpc: dial tcp 193.149.164.71:4310: i/o timeout
May 14 13:37:27.674997264 (330h49m38.195821009s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12aVyaTYczsdh8UPxFVHrcKjxHxz75b3Q67BcivVDLtM3beaSLm, address:193.149.164.70:4108): protocol: write tcp 188.166.225.170:41018->193.149.164.70:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.675143844 (330h49m38.195967589s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12UEesqk1Bez6iG4eUgV5Kpy1mM2zFmA3bVgCU4w5jF4KuX8DQP, address:193.149.164.68:4060): protocol: rpc: dial tcp 193.149.164.68:4060: i/o timeout; context deadline exceeded
May 14 13:37:27.675178129 (330h49m38.196001874s) [seg_pos={0 0}, upload=316975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
May 14 13:37:27.920638022 (330h49m38.441461777s) [upload=316975]: Done uploading segments...
May 14 13:37:27.920680102 (330h49m38.441503907s) [upload=316975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 8.478743926s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [X..................XX............X..........................] duration:
6.756422346s trace:
https://jaeger.dev.storj.io/trace/73b73ed5bcf2ff43
FAIL (2026-05-14 14:03:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.756445069s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............X......X.....................................] duration:
6.315275759s trace:
https://jaeger.dev.storj.io/trace/6bcf8e4c5b0862a8
FAIL (2026-05-14 14:03:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
May 14 13:36:28.324422154 (330h48m38.845245899s) [upload=316975]: Starting upload
May 14 13:36:28.324487729 (330h48m38.845311474s) [upload=316975]: Done starting upload
May 14 13:36:28.324543405 (330h48m38.84536714s) [upload=316975]: Uploading segments...
May 14 13:36:28.324757794 (330h48m38.845581539s) [upload=316975]: Got next segment. Inline: false
May 14 13:36:28.324788633 (330h48m38.845612368s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 14 13:36:28.324792991 (330h48m38.845616726s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 14 13:36:28.567718597 (330h48m39.088542342s) [seg_pos={0 0}, upload=316975]: Begin upload segment...
May 14 13:36:28.567770777 (330h48m39.088594522s) [seg_pos={0 0}, upload=316975]: Done begin upload segment.
May 14 13:36:28.567794703 (330h48m39.088618438s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.567819981 (330h48m39.088643726s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568338951 (330h48m39.089162696s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568829707 (330h48m39.089653452s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.569727691 (330h48m39.090551436s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570335932 (330h48m39.091159677s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570398291 (330h48m39.091222026s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570698164 (330h48m39.091521899s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570954873 (330h48m39.091778618s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571296846 (330h48m39.092120581s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571662814 (330h48m39.092486549s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572289079 (330h48m39.093112825s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572752583 (330h48m39.093576329s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573191752 (330h48m39.094015497s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573295591 (330h48m39.094119336s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573700533 (330h48m39.094524288s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:29.885678690 (330h48m40.406502435s) [upload=316975]: Next returned nil segment
May 14 13:36:29.885693760 (330h48m40.406517505s) [upload=316975]: Waiting for error group managing segments...
May 14 13:36:30.267895418 (330h48m40.788719153s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.436072812 (330h48m40.956896557s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.463303156 (330h48m40.984126901s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.495900862 (330h48m41.016724608s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.545724080 (330h48m41.066547825s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.796939386 (330h48m41.317763121s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.820623042 (330h48m41.341446847s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.914048005 (330h48m41.43487174s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:31.417499576 (330h48m41.938323331s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:32.451371893 (330h48m42.972195648s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:37:27.674200183 (330h49m38.195023918s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12LPgzYrzGJTay9LE1ZNm3dv9vhtj4qQqK5gMdc8B9nyBXdYLuw, address:193.149.164.67:4015): protocol: rpc: dial tcp 193.149.164.67:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674447916 (330h49m38.195271701s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1xK3trpsZJrYRGZteFknm89ykujQsEnjdYqTXYbWJEK4yp8Wio, address:193.149.164.69:4070): protocol: rpc: dial tcp 193.149.164.69:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674557244 (330h49m38.195380989s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12sCyZhtEDaFcQfcCsETubryphHfHcWzYay5pYMP5xhCuzTUaSd, address:193.149.164.66:4988): protocol: rpc: dial tcp 193.149.164.66:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674850474 (330h49m38.195674219s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1Ce95mJEDhVmLMjxKY6giHtJDG9gduHXDoNmvZRpmFPAZZTmC2, address:193.149.164.71:4310): rpc: dial tcp 193.149.164.71:4310: i/o timeout
May 14 13:37:27.674997264 (330h49m38.195821009s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12aVyaTYczsdh8UPxFVHrcKjxHxz75b3Q67BcivVDLtM3beaSLm, address:193.149.164.70:4108): protocol: write tcp 188.166.225.170:41018->193.149.164.70:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.675143844 (330h49m38.195967589s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12UEesqk1Bez6iG4eUgV5Kpy1mM2zFmA3bVgCU4w5jF4KuX8DQP, address:193.149.164.68:4060): protocol: rpc: dial tcp 193.149.164.68:4060: i/o timeout; context deadline exceeded
May 14 13:37:27.675178129 (330h49m38.196001874s) [seg_pos={0 0}, upload=316975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
May 14 13:37:27.920638022 (330h49m38.441461777s) [upload=316975]: Done uploading segments...
May 14 13:37:27.920680102 (330h49m38.441503907s) [upload=316975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 6.315300166s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [..............XXXXX....X.............X.X....................] duration:
3.850399199s trace:
https://jaeger.dev.storj.io/trace/273a3254470045aa
FAIL (2026-05-14 14:03:35 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.850415941s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [................X......X....................................] duration:
7.13257559s trace:
https://jaeger.dev.storj.io/trace/6bcf8e4c5b0862a8
FAIL (2026-05-14 14:03:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
May 14 13:36:28.324422154 (330h48m38.845245899s) [upload=316975]: Starting upload
May 14 13:36:28.324487729 (330h48m38.845311474s) [upload=316975]: Done starting upload
May 14 13:36:28.324543405 (330h48m38.84536714s) [upload=316975]: Uploading segments...
May 14 13:36:28.324757794 (330h48m38.845581539s) [upload=316975]: Got next segment. Inline: false
May 14 13:36:28.324788633 (330h48m38.845612368s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 14 13:36:28.324792991 (330h48m38.845616726s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 14 13:36:28.567718597 (330h48m39.088542342s) [seg_pos={0 0}, upload=316975]: Begin upload segment...
May 14 13:36:28.567770777 (330h48m39.088594522s) [seg_pos={0 0}, upload=316975]: Done begin upload segment.
May 14 13:36:28.567794703 (330h48m39.088618438s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.567819981 (330h48m39.088643726s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568338951 (330h48m39.089162696s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568829707 (330h48m39.089653452s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.569727691 (330h48m39.090551436s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570335932 (330h48m39.091159677s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570398291 (330h48m39.091222026s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570698164 (330h48m39.091521899s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570954873 (330h48m39.091778618s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571296846 (330h48m39.092120581s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571662814 (330h48m39.092486549s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572289079 (330h48m39.093112825s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572752583 (330h48m39.093576329s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573191752 (330h48m39.094015497s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573295591 (330h48m39.094119336s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573700533 (330h48m39.094524288s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:29.885678690 (330h48m40.406502435s) [upload=316975]: Next returned nil segment
May 14 13:36:29.885693760 (330h48m40.406517505s) [upload=316975]: Waiting for error group managing segments...
May 14 13:36:30.267895418 (330h48m40.788719153s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.436072812 (330h48m40.956896557s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.463303156 (330h48m40.984126901s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.495900862 (330h48m41.016724608s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.545724080 (330h48m41.066547825s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.796939386 (330h48m41.317763121s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.820623042 (330h48m41.341446847s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.914048005 (330h48m41.43487174s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:31.417499576 (330h48m41.938323331s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:32.451371893 (330h48m42.972195648s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:37:27.674200183 (330h49m38.195023918s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12LPgzYrzGJTay9LE1ZNm3dv9vhtj4qQqK5gMdc8B9nyBXdYLuw, address:193.149.164.67:4015): protocol: rpc: dial tcp 193.149.164.67:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674447916 (330h49m38.195271701s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1xK3trpsZJrYRGZteFknm89ykujQsEnjdYqTXYbWJEK4yp8Wio, address:193.149.164.69:4070): protocol: rpc: dial tcp 193.149.164.69:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674557244 (330h49m38.195380989s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12sCyZhtEDaFcQfcCsETubryphHfHcWzYay5pYMP5xhCuzTUaSd, address:193.149.164.66:4988): protocol: rpc: dial tcp 193.149.164.66:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674850474 (330h49m38.195674219s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1Ce95mJEDhVmLMjxKY6giHtJDG9gduHXDoNmvZRpmFPAZZTmC2, address:193.149.164.71:4310): rpc: dial tcp 193.149.164.71:4310: i/o timeout
May 14 13:37:27.674997264 (330h49m38.195821009s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12aVyaTYczsdh8UPxFVHrcKjxHxz75b3Q67BcivVDLtM3beaSLm, address:193.149.164.70:4108): protocol: write tcp 188.166.225.170:41018->193.149.164.70:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.675143844 (330h49m38.195967589s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12UEesqk1Bez6iG4eUgV5Kpy1mM2zFmA3bVgCU4w5jF4KuX8DQP, address:193.149.164.68:4060): protocol: rpc: dial tcp 193.149.164.68:4060: i/o timeout; context deadline exceeded
May 14 13:37:27.675178129 (330h49m38.196001874s) [seg_pos={0 0}, upload=316975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
May 14 13:37:27.920638022 (330h49m38.441461777s) [upload=316975]: Done uploading segments...
May 14 13:37:27.920680102 (330h49m38.441503907s) [upload=316975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 7.132615395s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.X..................XX............X.........................] duration:
6.897918462s trace:
https://jaeger.dev.storj.io/trace/73b73ed5bcf2ff43
FAIL (2026-05-14 14:03:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.897997522s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [...............XXXXX....X.............X.X...................] duration:
3.733275215s trace:
https://jaeger.dev.storj.io/trace/273a3254470045aa
FAIL (2026-05-14 14:02:48 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.733313239s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................X......X...................................] duration:
6.623031792s trace:
https://jaeger.dev.storj.io/trace/6bcf8e4c5b0862a8
FAIL (2026-05-14 14:02:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
May 14 13:36:28.324422154 (330h48m38.845245899s) [upload=316975]: Starting upload
May 14 13:36:28.324487729 (330h48m38.845311474s) [upload=316975]: Done starting upload
May 14 13:36:28.324543405 (330h48m38.84536714s) [upload=316975]: Uploading segments...
May 14 13:36:28.324757794 (330h48m38.845581539s) [upload=316975]: Got next segment. Inline: false
May 14 13:36:28.324788633 (330h48m38.845612368s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 14 13:36:28.324792991 (330h48m38.845616726s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 14 13:36:28.567718597 (330h48m39.088542342s) [seg_pos={0 0}, upload=316975]: Begin upload segment...
May 14 13:36:28.567770777 (330h48m39.088594522s) [seg_pos={0 0}, upload=316975]: Done begin upload segment.
May 14 13:36:28.567794703 (330h48m39.088618438s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.567819981 (330h48m39.088643726s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568338951 (330h48m39.089162696s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568829707 (330h48m39.089653452s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.569727691 (330h48m39.090551436s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570335932 (330h48m39.091159677s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570398291 (330h48m39.091222026s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570698164 (330h48m39.091521899s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570954873 (330h48m39.091778618s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571296846 (330h48m39.092120581s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571662814 (330h48m39.092486549s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572289079 (330h48m39.093112825s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572752583 (330h48m39.093576329s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573191752 (330h48m39.094015497s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573295591 (330h48m39.094119336s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573700533 (330h48m39.094524288s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:29.885678690 (330h48m40.406502435s) [upload=316975]: Next returned nil segment
May 14 13:36:29.885693760 (330h48m40.406517505s) [upload=316975]: Waiting for error group managing segments...
May 14 13:36:30.267895418 (330h48m40.788719153s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.436072812 (330h48m40.956896557s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.463303156 (330h48m40.984126901s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.495900862 (330h48m41.016724608s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.545724080 (330h48m41.066547825s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.796939386 (330h48m41.317763121s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.820623042 (330h48m41.341446847s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.914048005 (330h48m41.43487174s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:31.417499576 (330h48m41.938323331s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:32.451371893 (330h48m42.972195648s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:37:27.674200183 (330h49m38.195023918s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12LPgzYrzGJTay9LE1ZNm3dv9vhtj4qQqK5gMdc8B9nyBXdYLuw, address:193.149.164.67:4015): protocol: rpc: dial tcp 193.149.164.67:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674447916 (330h49m38.195271701s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1xK3trpsZJrYRGZteFknm89ykujQsEnjdYqTXYbWJEK4yp8Wio, address:193.149.164.69:4070): protocol: rpc: dial tcp 193.149.164.69:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674557244 (330h49m38.195380989s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12sCyZhtEDaFcQfcCsETubryphHfHcWzYay5pYMP5xhCuzTUaSd, address:193.149.164.66:4988): protocol: rpc: dial tcp 193.149.164.66:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674850474 (330h49m38.195674219s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1Ce95mJEDhVmLMjxKY6giHtJDG9gduHXDoNmvZRpmFPAZZTmC2, address:193.149.164.71:4310): rpc: dial tcp 193.149.164.71:4310: i/o timeout
May 14 13:37:27.674997264 (330h49m38.195821009s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12aVyaTYczsdh8UPxFVHrcKjxHxz75b3Q67BcivVDLtM3beaSLm, address:193.149.164.70:4108): protocol: write tcp 188.166.225.170:41018->193.149.164.70:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.675143844 (330h49m38.195967589s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12UEesqk1Bez6iG4eUgV5Kpy1mM2zFmA3bVgCU4w5jF4KuX8DQP, address:193.149.164.68:4060): protocol: rpc: dial tcp 193.149.164.68:4060: i/o timeout; context deadline exceeded
May 14 13:37:27.675178129 (330h49m38.196001874s) [seg_pos={0 0}, upload=316975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
May 14 13:37:27.920638022 (330h49m38.441461777s) [upload=316975]: Done uploading segments...
May 14 13:37:27.920680102 (330h49m38.441503907s) [upload=316975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 6.623056268s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................X......X..................................] duration:
8.069063864s trace:
https://jaeger.dev.storj.io/trace/6bcf8e4c5b0862a8
FAIL (2026-05-14 14:02:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
May 14 13:36:28.324422154 (330h48m38.845245899s) [upload=316975]: Starting upload
May 14 13:36:28.324487729 (330h48m38.845311474s) [upload=316975]: Done starting upload
May 14 13:36:28.324543405 (330h48m38.84536714s) [upload=316975]: Uploading segments...
May 14 13:36:28.324757794 (330h48m38.845581539s) [upload=316975]: Got next segment. Inline: false
May 14 13:36:28.324788633 (330h48m38.845612368s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 14 13:36:28.324792991 (330h48m38.845616726s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 14 13:36:28.567718597 (330h48m39.088542342s) [seg_pos={0 0}, upload=316975]: Begin upload segment...
May 14 13:36:28.567770777 (330h48m39.088594522s) [seg_pos={0 0}, upload=316975]: Done begin upload segment.
May 14 13:36:28.567794703 (330h48m39.088618438s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.567819981 (330h48m39.088643726s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568338951 (330h48m39.089162696s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568829707 (330h48m39.089653452s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.569727691 (330h48m39.090551436s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570335932 (330h48m39.091159677s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570398291 (330h48m39.091222026s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570698164 (330h48m39.091521899s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570954873 (330h48m39.091778618s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571296846 (330h48m39.092120581s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571662814 (330h48m39.092486549s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572289079 (330h48m39.093112825s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572752583 (330h48m39.093576329s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573191752 (330h48m39.094015497s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573295591 (330h48m39.094119336s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573700533 (330h48m39.094524288s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:29.885678690 (330h48m40.406502435s) [upload=316975]: Next returned nil segment
May 14 13:36:29.885693760 (330h48m40.406517505s) [upload=316975]: Waiting for error group managing segments...
May 14 13:36:30.267895418 (330h48m40.788719153s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.436072812 (330h48m40.956896557s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.463303156 (330h48m40.984126901s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.495900862 (330h48m41.016724608s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.545724080 (330h48m41.066547825s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.796939386 (330h48m41.317763121s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.820623042 (330h48m41.341446847s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.914048005 (330h48m41.43487174s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:31.417499576 (330h48m41.938323331s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:32.451371893 (330h48m42.972195648s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:37:27.674200183 (330h49m38.195023918s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12LPgzYrzGJTay9LE1ZNm3dv9vhtj4qQqK5gMdc8B9nyBXdYLuw, address:193.149.164.67:4015): protocol: rpc: dial tcp 193.149.164.67:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674447916 (330h49m38.195271701s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1xK3trpsZJrYRGZteFknm89ykujQsEnjdYqTXYbWJEK4yp8Wio, address:193.149.164.69:4070): protocol: rpc: dial tcp 193.149.164.69:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674557244 (330h49m38.195380989s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12sCyZhtEDaFcQfcCsETubryphHfHcWzYay5pYMP5xhCuzTUaSd, address:193.149.164.66:4988): protocol: rpc: dial tcp 193.149.164.66:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674850474 (330h49m38.195674219s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1Ce95mJEDhVmLMjxKY6giHtJDG9gduHXDoNmvZRpmFPAZZTmC2, address:193.149.164.71:4310): rpc: dial tcp 193.149.164.71:4310: i/o timeout
May 14 13:37:27.674997264 (330h49m38.195821009s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12aVyaTYczsdh8UPxFVHrcKjxHxz75b3Q67BcivVDLtM3beaSLm, address:193.149.164.70:4108): protocol: write tcp 188.166.225.170:41018->193.149.164.70:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.675143844 (330h49m38.195967589s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12UEesqk1Bez6iG4eUgV5Kpy1mM2zFmA3bVgCU4w5jF4KuX8DQP, address:193.149.164.68:4060): protocol: rpc: dial tcp 193.149.164.68:4060: i/o timeout; context deadline exceeded
May 14 13:37:27.675178129 (330h49m38.196001874s) [seg_pos={0 0}, upload=316975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
May 14 13:37:27.920638022 (330h49m38.441461777s) [upload=316975]: Done uploading segments...
May 14 13:37:27.920680102 (330h49m38.441503907s) [upload=316975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 8.069088981s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..X..................XX............X........................] duration:
6.653386656s trace:
https://jaeger.dev.storj.io/trace/73b73ed5bcf2ff43
FAIL (2026-05-14 14:02:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.653410412s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [................XXXXX....X.............X.X..................] duration:
4.00625376s trace:
https://jaeger.dev.storj.io/trace/273a3254470045aa
FAIL (2026-05-14 14:01:42 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 4.006279629s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [...X..................XX............X.......................] duration:
6.813941968s trace:
https://jaeger.dev.storj.io/trace/73b73ed5bcf2ff43
FAIL (2026-05-14 14:01:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.813965652s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................X......X.................................] duration:
6.26961516s trace:
https://jaeger.dev.storj.io/trace/6bcf8e4c5b0862a8
FAIL (2026-05-14 14:01:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
May 14 13:36:28.324422154 (330h48m38.845245899s) [upload=316975]: Starting upload
May 14 13:36:28.324487729 (330h48m38.845311474s) [upload=316975]: Done starting upload
May 14 13:36:28.324543405 (330h48m38.84536714s) [upload=316975]: Uploading segments...
May 14 13:36:28.324757794 (330h48m38.845581539s) [upload=316975]: Got next segment. Inline: false
May 14 13:36:28.324788633 (330h48m38.845612368s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 14 13:36:28.324792991 (330h48m38.845616726s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 14 13:36:28.567718597 (330h48m39.088542342s) [seg_pos={0 0}, upload=316975]: Begin upload segment...
May 14 13:36:28.567770777 (330h48m39.088594522s) [seg_pos={0 0}, upload=316975]: Done begin upload segment.
May 14 13:36:28.567794703 (330h48m39.088618438s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.567819981 (330h48m39.088643726s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568338951 (330h48m39.089162696s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568829707 (330h48m39.089653452s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.569727691 (330h48m39.090551436s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570335932 (330h48m39.091159677s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570398291 (330h48m39.091222026s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570698164 (330h48m39.091521899s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570954873 (330h48m39.091778618s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571296846 (330h48m39.092120581s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571662814 (330h48m39.092486549s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572289079 (330h48m39.093112825s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572752583 (330h48m39.093576329s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573191752 (330h48m39.094015497s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573295591 (330h48m39.094119336s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573700533 (330h48m39.094524288s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:29.885678690 (330h48m40.406502435s) [upload=316975]: Next returned nil segment
May 14 13:36:29.885693760 (330h48m40.406517505s) [upload=316975]: Waiting for error group managing segments...
May 14 13:36:30.267895418 (330h48m40.788719153s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.436072812 (330h48m40.956896557s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.463303156 (330h48m40.984126901s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.495900862 (330h48m41.016724608s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.545724080 (330h48m41.066547825s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.796939386 (330h48m41.317763121s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.820623042 (330h48m41.341446847s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.914048005 (330h48m41.43487174s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:31.417499576 (330h48m41.938323331s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:32.451371893 (330h48m42.972195648s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:37:27.674200183 (330h49m38.195023918s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12LPgzYrzGJTay9LE1ZNm3dv9vhtj4qQqK5gMdc8B9nyBXdYLuw, address:193.149.164.67:4015): protocol: rpc: dial tcp 193.149.164.67:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674447916 (330h49m38.195271701s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1xK3trpsZJrYRGZteFknm89ykujQsEnjdYqTXYbWJEK4yp8Wio, address:193.149.164.69:4070): protocol: rpc: dial tcp 193.149.164.69:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674557244 (330h49m38.195380989s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12sCyZhtEDaFcQfcCsETubryphHfHcWzYay5pYMP5xhCuzTUaSd, address:193.149.164.66:4988): protocol: rpc: dial tcp 193.149.164.66:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674850474 (330h49m38.195674219s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1Ce95mJEDhVmLMjxKY6giHtJDG9gduHXDoNmvZRpmFPAZZTmC2, address:193.149.164.71:4310): rpc: dial tcp 193.149.164.71:4310: i/o timeout
May 14 13:37:27.674997264 (330h49m38.195821009s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12aVyaTYczsdh8UPxFVHrcKjxHxz75b3Q67BcivVDLtM3beaSLm, address:193.149.164.70:4108): protocol: write tcp 188.166.225.170:41018->193.149.164.70:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.675143844 (330h49m38.195967589s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12UEesqk1Bez6iG4eUgV5Kpy1mM2zFmA3bVgCU4w5jF4KuX8DQP, address:193.149.164.68:4060): protocol: rpc: dial tcp 193.149.164.68:4060: i/o timeout; context deadline exceeded
May 14 13:37:27.675178129 (330h49m38.196001874s) [seg_pos={0 0}, upload=316975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
May 14 13:37:27.920638022 (330h49m38.441461777s) [upload=316975]: Done uploading segments...
May 14 13:37:27.920680102 (330h49m38.441503907s) [upload=316975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 6.26964111s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [....X..................XX............X......................] duration:
6.899970259s trace:
https://jaeger.dev.storj.io/trace/73b73ed5bcf2ff43
FAIL (2026-05-14 14:00:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.899996029s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [.................XXXXX....X.............X.X.................] duration:
3.906007085s trace:
https://jaeger.dev.storj.io/trace/273a3254470045aa
FAIL (2026-05-14 14:00:40 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.90603104s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................X......X................................] duration:
6.861106773s trace:
https://jaeger.dev.storj.io/trace/6bcf8e4c5b0862a8
FAIL (2026-05-14 14:00:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
May 14 13:36:28.324422154 (330h48m38.845245899s) [upload=316975]: Starting upload
May 14 13:36:28.324487729 (330h48m38.845311474s) [upload=316975]: Done starting upload
May 14 13:36:28.324543405 (330h48m38.84536714s) [upload=316975]: Uploading segments...
May 14 13:36:28.324757794 (330h48m38.845581539s) [upload=316975]: Got next segment. Inline: false
May 14 13:36:28.324788633 (330h48m38.845612368s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 14 13:36:28.324792991 (330h48m38.845616726s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 14 13:36:28.567718597 (330h48m39.088542342s) [seg_pos={0 0}, upload=316975]: Begin upload segment...
May 14 13:36:28.567770777 (330h48m39.088594522s) [seg_pos={0 0}, upload=316975]: Done begin upload segment.
May 14 13:36:28.567794703 (330h48m39.088618438s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.567819981 (330h48m39.088643726s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568338951 (330h48m39.089162696s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568829707 (330h48m39.089653452s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.569727691 (330h48m39.090551436s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570335932 (330h48m39.091159677s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570398291 (330h48m39.091222026s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570698164 (330h48m39.091521899s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570954873 (330h48m39.091778618s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571296846 (330h48m39.092120581s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571662814 (330h48m39.092486549s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572289079 (330h48m39.093112825s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572752583 (330h48m39.093576329s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573191752 (330h48m39.094015497s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573295591 (330h48m39.094119336s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573700533 (330h48m39.094524288s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:29.885678690 (330h48m40.406502435s) [upload=316975]: Next returned nil segment
May 14 13:36:29.885693760 (330h48m40.406517505s) [upload=316975]: Waiting for error group managing segments...
May 14 13:36:30.267895418 (330h48m40.788719153s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.436072812 (330h48m40.956896557s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.463303156 (330h48m40.984126901s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.495900862 (330h48m41.016724608s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.545724080 (330h48m41.066547825s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.796939386 (330h48m41.317763121s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.820623042 (330h48m41.341446847s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.914048005 (330h48m41.43487174s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:31.417499576 (330h48m41.938323331s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:32.451371893 (330h48m42.972195648s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:37:27.674200183 (330h49m38.195023918s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12LPgzYrzGJTay9LE1ZNm3dv9vhtj4qQqK5gMdc8B9nyBXdYLuw, address:193.149.164.67:4015): protocol: rpc: dial tcp 193.149.164.67:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674447916 (330h49m38.195271701s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1xK3trpsZJrYRGZteFknm89ykujQsEnjdYqTXYbWJEK4yp8Wio, address:193.149.164.69:4070): protocol: rpc: dial tcp 193.149.164.69:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674557244 (330h49m38.195380989s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12sCyZhtEDaFcQfcCsETubryphHfHcWzYay5pYMP5xhCuzTUaSd, address:193.149.164.66:4988): protocol: rpc: dial tcp 193.149.164.66:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674850474 (330h49m38.195674219s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1Ce95mJEDhVmLMjxKY6giHtJDG9gduHXDoNmvZRpmFPAZZTmC2, address:193.149.164.71:4310): rpc: dial tcp 193.149.164.71:4310: i/o timeout
May 14 13:37:27.674997264 (330h49m38.195821009s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12aVyaTYczsdh8UPxFVHrcKjxHxz75b3Q67BcivVDLtM3beaSLm, address:193.149.164.70:4108): protocol: write tcp 188.166.225.170:41018->193.149.164.70:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.675143844 (330h49m38.195967589s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12UEesqk1Bez6iG4eUgV5Kpy1mM2zFmA3bVgCU4w5jF4KuX8DQP, address:193.149.164.68:4060): protocol: rpc: dial tcp 193.149.164.68:4060: i/o timeout; context deadline exceeded
May 14 13:37:27.675178129 (330h49m38.196001874s) [seg_pos={0 0}, upload=316975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
May 14 13:37:27.920638022 (330h49m38.441461777s) [upload=316975]: Done uploading segments...
May 14 13:37:27.920680102 (330h49m38.441503907s) [upload=316975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 6.861125649s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................X......X...............................] duration:
7.58444892s trace:
https://jaeger.dev.storj.io/trace/6bcf8e4c5b0862a8
FAIL (2026-05-14 14:00:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
May 14 13:36:28.324422154 (330h48m38.845245899s) [upload=316975]: Starting upload
May 14 13:36:28.324487729 (330h48m38.845311474s) [upload=316975]: Done starting upload
May 14 13:36:28.324543405 (330h48m38.84536714s) [upload=316975]: Uploading segments...
May 14 13:36:28.324757794 (330h48m38.845581539s) [upload=316975]: Got next segment. Inline: false
May 14 13:36:28.324788633 (330h48m38.845612368s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 14 13:36:28.324792991 (330h48m38.845616726s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 14 13:36:28.567718597 (330h48m39.088542342s) [seg_pos={0 0}, upload=316975]: Begin upload segment...
May 14 13:36:28.567770777 (330h48m39.088594522s) [seg_pos={0 0}, upload=316975]: Done begin upload segment.
May 14 13:36:28.567794703 (330h48m39.088618438s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.567819981 (330h48m39.088643726s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568338951 (330h48m39.089162696s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568829707 (330h48m39.089653452s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.569727691 (330h48m39.090551436s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570335932 (330h48m39.091159677s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570398291 (330h48m39.091222026s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570698164 (330h48m39.091521899s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570954873 (330h48m39.091778618s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571296846 (330h48m39.092120581s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571662814 (330h48m39.092486549s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572289079 (330h48m39.093112825s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572752583 (330h48m39.093576329s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573191752 (330h48m39.094015497s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573295591 (330h48m39.094119336s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573700533 (330h48m39.094524288s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:29.885678690 (330h48m40.406502435s) [upload=316975]: Next returned nil segment
May 14 13:36:29.885693760 (330h48m40.406517505s) [upload=316975]: Waiting for error group managing segments...
May 14 13:36:30.267895418 (330h48m40.788719153s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.436072812 (330h48m40.956896557s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.463303156 (330h48m40.984126901s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.495900862 (330h48m41.016724608s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.545724080 (330h48m41.066547825s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.796939386 (330h48m41.317763121s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.820623042 (330h48m41.341446847s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.914048005 (330h48m41.43487174s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:31.417499576 (330h48m41.938323331s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:32.451371893 (330h48m42.972195648s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:37:27.674200183 (330h49m38.195023918s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12LPgzYrzGJTay9LE1ZNm3dv9vhtj4qQqK5gMdc8B9nyBXdYLuw, address:193.149.164.67:4015): protocol: rpc: dial tcp 193.149.164.67:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674447916 (330h49m38.195271701s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1xK3trpsZJrYRGZteFknm89ykujQsEnjdYqTXYbWJEK4yp8Wio, address:193.149.164.69:4070): protocol: rpc: dial tcp 193.149.164.69:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674557244 (330h49m38.195380989s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12sCyZhtEDaFcQfcCsETubryphHfHcWzYay5pYMP5xhCuzTUaSd, address:193.149.164.66:4988): protocol: rpc: dial tcp 193.149.164.66:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674850474 (330h49m38.195674219s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1Ce95mJEDhVmLMjxKY6giHtJDG9gduHXDoNmvZRpmFPAZZTmC2, address:193.149.164.71:4310): rpc: dial tcp 193.149.164.71:4310: i/o timeout
May 14 13:37:27.674997264 (330h49m38.195821009s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12aVyaTYczsdh8UPxFVHrcKjxHxz75b3Q67BcivVDLtM3beaSLm, address:193.149.164.70:4108): protocol: write tcp 188.166.225.170:41018->193.149.164.70:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.675143844 (330h49m38.195967589s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12UEesqk1Bez6iG4eUgV5Kpy1mM2zFmA3bVgCU4w5jF4KuX8DQP, address:193.149.164.68:4060): protocol: rpc: dial tcp 193.149.164.68:4060: i/o timeout; context deadline exceeded
May 14 13:37:27.675178129 (330h49m38.196001874s) [seg_pos={0 0}, upload=316975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
May 14 13:37:27.920638022 (330h49m38.441461777s) [upload=316975]: Done uploading segments...
May 14 13:37:27.920680102 (330h49m38.441503907s) [upload=316975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 7.584475571s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [..................XXXXX....X.............X.X................] duration:
4.369674743s trace:
https://jaeger.dev.storj.io/trace/273a3254470045aa
FAIL (2026-05-14 14:00:17 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 4.369732082s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.....X..................XX............X.....................] duration:
6.911937399s trace:
https://jaeger.dev.storj.io/trace/73b73ed5bcf2ff43
FAIL (2026-05-14 13:59:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.911956314s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [...................XXXXX....X.............X.X...............] duration:
4.181925717s trace:
https://jaeger.dev.storj.io/trace/273a3254470045aa
FAIL (2026-05-14 13:59:38 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 4.18194832s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [....................XXXXX....X.............X.X..............] duration:
4.086381318s trace:
https://jaeger.dev.storj.io/trace/273a3254470045aa
FAIL (2026-05-14 13:59:20 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 4.086408341s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................X......X..............................] duration:
6.324643598s trace:
https://jaeger.dev.storj.io/trace/6bcf8e4c5b0862a8
FAIL (2026-05-14 13:59:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
May 14 13:36:28.324422154 (330h48m38.845245899s) [upload=316975]: Starting upload
May 14 13:36:28.324487729 (330h48m38.845311474s) [upload=316975]: Done starting upload
May 14 13:36:28.324543405 (330h48m38.84536714s) [upload=316975]: Uploading segments...
May 14 13:36:28.324757794 (330h48m38.845581539s) [upload=316975]: Got next segment. Inline: false
May 14 13:36:28.324788633 (330h48m38.845612368s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 14 13:36:28.324792991 (330h48m38.845616726s) [upload=316975]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 14 13:36:28.567718597 (330h48m39.088542342s) [seg_pos={0 0}, upload=316975]: Begin upload segment...
May 14 13:36:28.567770777 (330h48m39.088594522s) [seg_pos={0 0}, upload=316975]: Done begin upload segment.
May 14 13:36:28.567794703 (330h48m39.088618438s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.567819981 (330h48m39.088643726s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568338951 (330h48m39.089162696s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.568829707 (330h48m39.089653452s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.569727691 (330h48m39.090551436s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570335932 (330h48m39.091159677s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570398291 (330h48m39.091222026s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570698164 (330h48m39.091521899s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.570954873 (330h48m39.091778618s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571296846 (330h48m39.092120581s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.571662814 (330h48m39.092486549s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572289079 (330h48m39.093112825s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.572752583 (330h48m39.093576329s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573191752 (330h48m39.094015497s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573295591 (330h48m39.094119336s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:28.573700533 (330h48m39.094524288s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Uploading piece...
May 14 13:36:29.885678690 (330h48m40.406502435s) [upload=316975]: Next returned nil segment
May 14 13:36:29.885693760 (330h48m40.406517505s) [upload=316975]: Waiting for error group managing segments...
May 14 13:36:30.267895418 (330h48m40.788719153s) [noise=true , address=167.17.72.69:4716 , piece_id=M6J6OVFWROXK2ZXBAUSUUZJIBGW4QP2NUPG5DSRDYJXIHTJX4T3Q, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.436072812 (330h48m40.956896557s) [noise=true , address=167.17.72.70:4762 , piece_id=3AXWO654HLI5AHEKIH4BR4EH6DQMLOIZR5MLY7JQYCGU6LGGRWYQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.463303156 (330h48m40.984126901s) [noise=true , address=145.79.227.17:4979 , piece_id=HLHTULWWCSGVL6D3VPPMLI5OPCO4YHLWAIBNREXTOEMPMISXX3QQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.495900862 (330h48m41.016724608s) [noise=true , address=77.90.31.10:4542 , piece_id=6CLU53SWUWW7FA5WWNZ6ZWCZYD4KHAP4NUJJXOQZOPTOPIP4H6OA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.545724080 (330h48m41.066547825s) [noise=true , address=77.90.31.12:4618 , piece_id=4R6C7VKGNHOSPBKEK6OJYR5PNC6AX4UBO7WWSANGMPZNWL4J4BEA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.796939386 (330h48m41.317763121s) [noise=true , address=193.149.164.78:4486 , piece_id=Y7PGNX37MO3WAMXLPRWW3IIMPXYVZPUEKHKD27LCKEI5XBJLCSBQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.820623042 (330h48m41.341446847s) [noise=true , address=167.17.72.71:4812 , piece_id=4OGK2UKIDBJC77LQFJGWIETDMSATWDLIOK66BYPMM7QJX3AW2K6A, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:30.914048005 (330h48m41.43487174s) [noise=true , address=167.17.72.68:4663 , piece_id=OWKX55OXLPEABO3YSWV4WPCEEQMYHYY7BIRIXDM2XBSAQEM6B6EQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:31.417499576 (330h48m41.938323331s) [noise=true , address=193.149.164.79:4517 , piece_id=I63XYMJ5KIW4KZRMZKMPKZS76X7HHL7S3BQX4PULEKU73UOPYWZA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:36:32.451371893 (330h48m42.972195648s) [noise=true , address=145.79.227.31:4816 , piece_id=VQBEZTKNZAX2X7ESTZOM2INQ3FENU7OPNCKLAEZ5UI52TXDUIGIA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: <nil>
May 14 13:37:27.674200183 (330h49m38.195023918s) [noise=true , address=193.149.164.67:4015 , piece_id=QQ6ASSWTQUEIXV6UHG2IGTP5ETBJAFHGBLD2YYZGHMPYOU67VMLA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12LPgzYrzGJTay9LE1ZNm3dv9vhtj4qQqK5gMdc8B9nyBXdYLuw, address:193.149.164.67:4015): protocol: rpc: dial tcp 193.149.164.67:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674447916 (330h49m38.195271701s) [noise=true , address=193.149.164.69:4070 , piece_id=5IXZR4L6IAX32TRWBFVABDQBSMHA36WNFGCSDBVWBEE5ZBGUR2LA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1xK3trpsZJrYRGZteFknm89ykujQsEnjdYqTXYbWJEK4yp8Wio, address:193.149.164.69:4070): protocol: rpc: dial tcp 193.149.164.69:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674557244 (330h49m38.195380989s) [noise=true , address=193.149.164.66:4988 , piece_id=B7CH3TDCEHFRGAZCWN3OCZJBIPGV2B3V2HC5AJLMLWCJJDR7C2FQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12sCyZhtEDaFcQfcCsETubryphHfHcWzYay5pYMP5xhCuzTUaSd, address:193.149.164.66:4988): protocol: rpc: dial tcp 193.149.164.66:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.674850474 (330h49m38.195674219s) [noise=true , address=193.149.164.71:4310 , piece_id=2OLPYNQE7X6HS4YFJAYEFUG7VVCZN3QV37PHZTJF6S3LREQ6LMQA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:1Ce95mJEDhVmLMjxKY6giHtJDG9gduHXDoNmvZRpmFPAZZTmC2, address:193.149.164.71:4310): rpc: dial tcp 193.149.164.71:4310: i/o timeout
May 14 13:37:27.674997264 (330h49m38.195821009s) [noise=true , address=193.149.164.70:4108 , piece_id=PKSTYTZFCNOXPAYVB6XOMT3LN5QTTX2QLSX3ECAGSYSBOLPIWSNA, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12aVyaTYczsdh8UPxFVHrcKjxHxz75b3Q67BcivVDLtM3beaSLm, address:193.149.164.70:4108): protocol: write tcp 188.166.225.170:41018->193.149.164.70:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
May 14 13:37:27.675143844 (330h49m38.195967589s) [noise=true , address=193.149.164.68:4060 , piece_id=WZDIN7U62DMQLPWRLGHEJAMU4POSK7QSDW3FX54CH3LVP7P2RMZQ, seg_pos={0 0}, upload=316975]: Done uploading piece. err: ecclient: upload failed (node:12UEesqk1Bez6iG4eUgV5Kpy1mM2zFmA3bVgCU4w5jF4KuX8DQP, address:193.149.164.68:4060): protocol: rpc: dial tcp 193.149.164.68:4060: i/o timeout; context deadline exceeded
May 14 13:37:27.675178129 (330h49m38.196001874s) [seg_pos={0 0}, upload=316975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
May 14 13:37:27.920638022 (330h49m38.441461777s) [upload=316975]: Done uploading segments...
May 14 13:37:27.920680102 (330h49m38.441503907s) [upload=316975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 6.324663795s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [......X..................XX............X....................] duration:
7.179064608s trace:
https://jaeger.dev.storj.io/trace/73b73ed5bcf2ff43
FAIL (2026-05-14 13:58:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.179085859s
core
uplink
us-select-1
us1-placement-20
|