Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.693438626s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:12:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.693465227s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.892880323s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:12:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.892909218s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.701392105s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:12:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.701421112s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.894460361s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:11:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.894492673s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.939105156s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:11:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.939129532s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.921958357s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:10:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.92198617s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.756251856s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:10:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.756270912s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.890063597s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:10:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.890159019s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.901226754s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:09:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.901255449s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.836906548s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:09:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.836930934s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.940013996s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:09:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.940056537s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.75344532s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:08:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.753470356s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.899499904s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:07:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.899521636s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.756537365s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:07:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.756554216s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.243731412s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:07:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.243757051s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000592427s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:07:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000616212s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.37938244s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 19:14:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.379414249s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.41750402s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 19:13:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.417530981s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.537883108s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 19:13:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.539207139s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.520289736s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 19:12:26 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520313852s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.544509669s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 19:11:55 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.544536249s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.605022404s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 19:11:05 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.605071317s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 4.236096343s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 19:10:23 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.236129857s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.372524309s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 19:09:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.372550971s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.447976589s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 19:08:30 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.447996898s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.535386062s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 19:07:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.535439483s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.483200187s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 19:06:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.483262526s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.389560236s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 19:05:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.389621302s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.498757313s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 19:04:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.498784274s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.390566302s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 19:03:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.390663667s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.563856891s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 19:02:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.563889193s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.47248618s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 19:02:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.472513903s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.562276584s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 19:02:01 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.562308656s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.497264911s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 19:01:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.497284277s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.658623515s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 19:00:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.65865265s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.507804468s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 18:59:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.50783166s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.569103415s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 18:59:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.569153751s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.576787016s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 18:58:13 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.576848795s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.386344071s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 18:57:00 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.386371023s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.526880376s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 18:56:01 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.526905796s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.505503762s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 18:55:43 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.505532707s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.499764008s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 18:54:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.499814405s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.445439253s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 18:53:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.445472837s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.474754885s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 18:53:13 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.474779523s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.54888869s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 18:52:50 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.548918437s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.444513907s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 18:52:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.444657992s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.546035284s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 18:51:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.546071344s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.526438133s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 18:50:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.526535066s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.484507408s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 18:49:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484558665s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.639479671s trace: https://jaeger.dev.storj.io/trace/256f88e6f0689c61 FAIL (2025-11-28 18:48:44 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.639528634s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)