Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 524409 reports (Page 5204 of 10489)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.96357675s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:49:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.963604051s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.396788331s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:49:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.396821794s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.729757147s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:49:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.729865544s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.989712242s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:49:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.989757349s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.789169569s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:49:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.789203684s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.458096808s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:49:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.458120995s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.389509022s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:49:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.38953474s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.792912537s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:48:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.792929921s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.339829695s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:48:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.339929616s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.996000479s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:48:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.996027571s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 3.641026002s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:48:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.641055219s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.693402541s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:48:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.693424262s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.714498443s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:48:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.714577182s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.923817449s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:47:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.923845072s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.940963501s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:47:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.940992095s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.429084251s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:47:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.42910464s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.090566417s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:47:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.090590442s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.896072977s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:47:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.896096762s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.790775147s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:46:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.790808s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.935948928s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:46:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.935977972s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.442609768s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:46:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.442638453s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.861288357s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:46:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.861317955s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.845548476s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:46:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.845574836s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 4.335694325s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:45:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.335729923s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.909888565s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:45:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.909915546s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.788003283s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:45:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.788032228s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.676634727s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:45:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.676684833s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.557546447s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:45:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.557574712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.906009257s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:44:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.90603136s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.070561257s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:44:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.070585533s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.900204019s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:44:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.900240378s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.462514239s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:44:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.462542473s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.853539383s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:44:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.853566154s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.745394795s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:44:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.745410816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.957910577s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:44:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.957942309s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.439860883s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:44:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.439889639s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.801564173s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:43:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.801592998s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.792844999s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:43:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.792874496s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.503772033s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:43:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.503800378s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.805622279s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:43:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.805651373s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.735580339s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:43:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.735614574s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.784961962s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:43:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.784990517s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.386973294s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:42:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.386995455s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.728337684s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:42:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.728368513s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.981182281s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:42:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.981213642s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.81543857s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:42:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.815469651s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.714963837s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:41:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.714993733s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.893131586s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:41:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.893158998s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.513756915s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:41:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.513788234s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.975530861s trace: https://jaeger.dev.storj.io/trace/2f8da4022f0e6d72 FAIL (2025-11-28 22:41:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:21.486649527 (470h3m20.038119989s) [upload=413926]: Starting upload Nov 28 22:07:21.486777562 (470h3m20.038248023s) [upload=413926]: Done starting upload Nov 28 22:07:21.487000386 (470h3m20.038470838s) [upload=413926]: Uploading segments... Nov 28 22:07:21.487113672 (470h3m20.038584123s) [upload=413926]: Got next segment. Inline: false Nov 28 22:07:21.487149410 (470h3m20.038619861s) [upload=413926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:21.487154620 (470h3m20.038625071s) [upload=413926]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:21.735855247 (470h3m20.287325698s) [seg_pos={0 0}, upload=413926]: Begin upload segment... Nov 28 22:07:21.735903108 (470h3m20.287373569s) [seg_pos={0 0}, upload=413926]: Done begin upload segment. Nov 28 22:07:21.735930951 (470h3m20.287401402s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.736867164 (470h3m20.288337625s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737387685 (470h3m20.288858136s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.737960135 (470h3m20.289430596s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.738439087 (470h3m20.289909548s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.739002229 (470h3m20.29047268s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.741405956 (470h3m20.292876417s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.742342961 (470h3m20.293813422s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.743211985 (470h3m20.294682436s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.744602123 (470h3m20.296072584s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.751769040 (470h3m20.303239501s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.757267199 (470h3m20.30873766s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.762188010 (470h3m20.313658481s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.764395414 (470h3m20.315865875s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.766080753 (470h3m20.317551214s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:21.767658637 (470h3m20.319129108s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Uploading piece... Nov 28 22:07:23.481506301 (470h3m22.032976762s) [upload=413926]: Next returned nil segment Nov 28 22:07:23.481521169 (470h3m22.03299162s) [upload=413926]: Waiting for error group managing segments... Nov 28 22:07:23.767697058 (470h3m22.31916751s) [noise=true , address=145.79.227.14:4839 , piece_id=ZOQNUGORLQCLGODR5RTHNXRMQKR5UXMNTIWLR57EVE4NQIEJUAKQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.771782027 (470h3m22.323252478s) [noise=true , address=145.79.227.9:4366 , piece_id=XW5267YDQIGWOT6SUGNUSKX55YHTMQY5GTOW33D2GKHEFOBR3LJA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.773982126 (470h3m22.325452587s) [noise=true , address=77.90.31.14:4662 , piece_id=CZFPDVKVEA5LCLMUU5IJ2Y3ZOS2AD2MVW3F7MGGP5AMDMXOHIBOA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.778457769 (470h3m22.32992822s) [noise=true , address=145.79.227.33:4870 , piece_id=YNHD6GFLS7HK44X4L3FP35ZI3SUHX27WZXIFUEZC6JL2YVZG7XVA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.784240660 (470h3m22.335711121s) [noise=true , address=145.79.227.2:4115 , piece_id=N27IRV6W7SLXPULFB6765EECPZHPYFAOQHCBXDEBEOPP6HASBW2A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.786064574 (470h3m22.337535035s) [noise=true , address=145.79.227.34:4905 , piece_id=FOGG6QVT3V6CMGPPDIWLBXJ5ZXPGGM7JDOT6HCNWW2FGXQOAMYUA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.795902407 (470h3m22.347372868s) [noise=true , address=145.79.227.42:4122 , piece_id=2Q7GP7KH4RAA5DWN3UGKI5UZUTSUVJSGFBHR4FCVTW6K7ODEKW3A, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.800759555 (470h3m22.352230016s) [noise=true , address=145.79.227.17:4972 , piece_id=IDNFZBXBN5X4ODAUOLI2LWPMJVZRY2M5EJI6RYXKSO5AQJONGDXQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.812276557 (470h3m22.363747008s) [noise=true , address=145.79.227.21:4473 , piece_id=DCYEGK3FXMYCCO65WGPLHNO5ENTTGKIY5F7ZGNGNDKGD537QFIEA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.813227167 (470h3m22.364697628s) [noise=true , address=145.79.227.16:4910 , piece_id=3EPBK74H77BF6FRZVTITAVCOUIWTAUXFXHPSJ2O7MALBVLRPAQIQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.842249682 (470h3m22.393720143s) [noise=true , address=145.79.227.6:4258 , piece_id=EXZ7CA3B45ALJHUSAX2WHKKEDK7HHG2A52DYW2JNRTRLLOEK3SBQ, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:23.918664091 (470h3m22.470134552s) [noise=true , address=77.90.31.17:4772 , piece_id=MKIGU2SPUZOMASMTWD3JGR6BG3OUKWTEWXMHYWOZKDCAQFBT243Q, seg_pos={0 0}, upload=413926]: Done uploading piece. err: <nil> Nov 28 22:07:50.788587419 (470h3m49.34005787s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4XA33MUQ424DG6LDV7OCUA2Q6OOQXWHSD3PSWKUTFPSKVS3UMTA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.13:4153: use of closed network connection Nov 28 22:07:50.788614880 (470h3m49.340085331s) [noise=true , address=45.91.116.6:4975 , piece_id=DEBJRB2Q4OYBT7QMITKW4L4J5XDYZ3ZQ3EKL4GNVKBIBAO4JYQMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:53582->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788746692 (470h3m49.340217243s) [noise=true , address=45.91.116.4:4938 , piece_id=ZPWTDVFUMNL3KGNBBP2B5KPZSMA4PRRR3UQYSTOLVTIDUZZXMDMA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:32874->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788830170 (470h3m49.340300631s) [noise=true , address=45.91.116.5:4299 , piece_id=XRG2MBKHNF7BEXRXITYODEDHXW6WQLGL3V7EPBPEOMJ7DH4CSHHA, seg_pos={0 0}, upload=413926]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:50354->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:07:50.788855919 (470h3m49.34032638s) [seg_pos={0 0}, upload=413926]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:07:51.029595799 (470h3m49.58106627s) [upload=413926]: Done uploading segments... Nov 28 22:07:51.029642828 (470h3m49.581113289s) [upload=413926]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.975564025s core uplink us-select-1

1
...
5202
5203
5204
5205
5206
...
10489