Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.797214428s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:25:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.797237461s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.909462523s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:25:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.909491019s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.980263882s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:24:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.980311153s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.884926861s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:24:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.884954474s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.871556369s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:24:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.871582509s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.624330016s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:23:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.624403226s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.85495877s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:23:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.854985851s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.131734497s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:23:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.131761938s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.859886304s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:22:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.859929905s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.780612974s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:22:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.78063679s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.771886701s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:22:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.771910945s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.828369281s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:21:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.828399178s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.669855364s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:21:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.669887865s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.845972467s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:21:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.846000181s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.828876305s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:20:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.828905741s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.04281557s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:20:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.042834094s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.990114138s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:20:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.990147912s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.760693343s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:19:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.760709975s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.046506859s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:19:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.046534903s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.78445387s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:19:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.784490299s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.71743317s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:18:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.717459781s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.062933245s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:18:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.062965155s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.960077072s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:17:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.960110486s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.722544737s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:17:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.722578422s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.854215045s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:17:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.854242366s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.629749889s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:16:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.629785998s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.267773347s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:16:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.267799497s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.751397661s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:16:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.751428059s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.316377944s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:15:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.316412601s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.96002752s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:15:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.960062006s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.962940615s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:15:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.962971203s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.940392816s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:14:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.940422742s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.09634578s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:14:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.096375508s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.771781394s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:14:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.771810511s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.830765665s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:14:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.830819778s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.794425435s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:13:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.794454581s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.881952137s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:13:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.881984449s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.693438626s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:12:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.693465227s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.892880323s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:12:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.892909218s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.701392105s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:12:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.701421112s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.894460361s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:11:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.894492673s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.939105156s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:11:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.939129532s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.921958357s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:10:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.92198617s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.756251856s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:10:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.756270912s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.890063597s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:10:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.890159019s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.901226754s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:09:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.901255449s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.836906548s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:09:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.836930934s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.940013996s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:09:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.940056537s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.75344532s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:08:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.753470356s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.899499904s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:07:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.899521636s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)