Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX.....X...XX] duration: 30.002584341s trace: https://jaeger.dev.storj.io/trace/5214518dc57f4d61 FAIL (2025-12-31 01:18:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pb.BatchRequestItem_ObjectBegin Dec 31 01:18:01.171218535 (609h49m1.04230158s) [upload=534427]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:18:01.420264420 (609h49m1.291347455s) [seg_pos={0 0}, upload=534427]: Begin upload segment... Dec 31 01:18:01.420326348 (609h49m1.291409393s) [seg_pos={0 0}, upload=534427]: Done begin upload segment. Dec 31 01:18:01.420371634 (609h49m1.291454679s) [noise=true , address=45.91.116.14:4189 , piece_id=OWINAXCRNMWKQVZTHGVW7BNQXO7CNSTZBRKC26CU5KHG6ARW5AEQ, seg_pos={0 0}, upload=534427]: Uploading piece... Dec 31 01:18:01.420416881 (609h49m1.291499916s) [noise=true , address=145.79.227.39:4014 , piece_id=FI66MADAOH7DWWCCHNSEG64MIUG6SUXVPRDCQPDQTMRMBXFLZGWQ, seg_pos={0 0}, upload=534427]: Uploading piece... Dec 31 01:18:01.420938955 (609h49m1.292022s) [noise=true , address=145.79.227.43:4159 , piece_id=AFKB2D5UHMYLSEJGKRD77WLTR625BOFOYLCGE2XERPZDQ3UUTPHQ, seg_pos={0 0}, upload=534427]: Uploading piece... Dec 31 01:18:01.421324469 (609h49m1.292407514s) [noise=true , address=45.91.116.7:4009 , piece_id=TOHJCZG6Q34RDX5ZTZFVJLWVJEF7M27WXQTQORUT3KXSXHQEOVWA, seg_pos={0 0}, upload=534427]: Uploading piece... Dec 31 01:18:01.421722095 (609h49m1.29280513s) [noise=true , address=77.90.31.12:4590 , piece_id=WUNBDHFAKDWGSMIKZPARJTDRWLFKLZIBTLB3X6K6OS3SNCP4B5RQ, seg_pos={0 0}, upload=534427]: Uploading piece... Dec 31 01:18:01.422109222 (609h49m1.293192268s) [noise=true , address=45.91.116.9:4262 , piece_id=XF6UPSTRGN2G7UMPNRVTRZB4HSQIPVOJZ6WVZA5PCTG5CHVSADDA, seg_pos={0 0}, upload=534427]: Uploading piece... Dec 31 01:18:01.422743159 (609h49m1.293826204s) [noise=true , address=145.79.227.13:4802 , piece_id=HYQ4PKXMUGRND3FBMJ4KVIWQET5KZUDMQC2HBKCSQG2STCP5JWYA, seg_pos={0 0}, upload=534427]: Uploading piece... Dec 31 01:18:01.422758649 (609h49m1.293841684s) [noise=true , address=145.79.227.25:4619 , piece_id=O4O5UKATE3GDV7XRVPOEQ7ZLHRZD6TCVFRWTCUZIFASKQOCA77GQ, seg_pos={0 0}, upload=534427]: Uploading piece... Dec 31 01:18:01.423226199 (609h49m1.294309244s) [noise=true , address=45.91.116.5:4299 , piece_id=MDJNFXBJHDNE4WPAZL2P25CKBQH5TNQ4SGLMB7BHJARE4HI26PPA, seg_pos={0 0}, upload=534427]: Uploading piece... Dec 31 01:18:01.423651068 (609h49m1.294734113s) [noise=true , address=145.79.227.38:4978 , piece_id=KHFQVGJUDBW44VD373WU5TVY53Q2DWWAZWHYEFKJ7WEZTGZZE32A, seg_pos={0 0}, upload=534427]: Uploading piece... Dec 31 01:18:01.424056270 (609h49m1.295139315s) [noise=true , address=145.79.227.22:4511 , piece_id=TGQERI26IYBZWKL5EH2VJO5G4BMA5WMZECYP4CBH54LG3UZGSVRQ, seg_pos={0 0}, upload=534427]: Uploading piece... Dec 31 01:18:01.424426344 (609h49m1.295509389s) [noise=true , address=145.79.227.8:4329 , piece_id=REQU7WRHQIPH5ZYHSXU3LQGXY3ZCZ5AL5KJVS6ZNJWSR3WTSGA7Q, seg_pos={0 0}, upload=534427]: Uploading piece... Dec 31 01:18:01.424791921 (609h49m1.295875116s) [noise=true , address=145.79.227.15:4875 , piece_id=52VZIYF6XEUDVRVPPJLH4KKAKP7NBG6T2K57HEYSWQU7GZ3JQPDQ, seg_pos={0 0}, upload=534427]: Uploading piece... Dec 31 01:18:01.426375716 (609h49m1.297458761s) [noise=true , address=77.90.31.16:4736 , piece_id=J5YEC3SGYIKLGYUSSQH5TNRY7JTBNYUIOJQVJROWHEFISF6ICZWA, seg_pos={0 0}, upload=534427]: Uploading piece... Dec 31 01:18:01.433162979 (609h49m1.304246024s) [noise=true , address=145.79.227.7:4295 , piece_id=4ZTDLM7OGMPW5AL76RLVE5P6NPKSV35YHUUQXG22GILM2KM3HF7Q, seg_pos={0 0}, upload=534427]: Uploading piece... Dec 31 01:18:01.439346083 (609h49m1.310429118s) [noise=true , address=77.90.31.19:4844 , piece_id=ZVLY6AIG2ZQQ5F6EUZYLKS7KWRZ3XW665IXSCL55JWBTSREXT42A, seg_pos={0 0}, upload=534427]: Uploading piece... Dec 31 01:18:03.119237392 (609h49m2.990320437s) [upload=534427]: Next returned nil segment Dec 31 01:18:03.119257661 (609h49m2.990340696s) [upload=534427]: Waiting for error group managing segments... Dec 31 01:18:03.418088812 (609h49m3.289171847s) [noise=true , address=145.79.227.22:4511 , piece_id=TGQERI26IYBZWKL5EH2VJO5G4BMA5WMZECYP4CBH54LG3UZGSVRQ, seg_pos={0 0}, upload=534427]: Done uploading piece. err: <nil> Dec 31 01:18:03.425442634 (609h49m3.296525679s) [noise=true , address=145.79.227.39:4014 , piece_id=FI66MADAOH7DWWCCHNSEG64MIUG6SUXVPRDCQPDQTMRMBXFLZGWQ, seg_pos={0 0}, upload=534427]: Done uploading piece. err: <nil> Dec 31 01:18:03.427759917 (609h49m3.298842962s) [noise=true , address=145.79.227.43:4159 , piece_id=AFKB2D5UHMYLSEJGKRD77WLTR625BOFOYLCGE2XERPZDQ3UUTPHQ, seg_pos={0 0}, upload=534427]: Done uploading piece. err: <nil> Dec 31 01:18:03.434786326 (609h49m3.305869361s) [noise=true , address=145.79.227.13:4802 , piece_id=HYQ4PKXMUGRND3FBMJ4KVIWQET5KZUDMQC2HBKCSQG2STCP5JWYA, seg_pos={0 0}, upload=534427]: Done uploading piece. err: <nil> Dec 31 01:18:03.434881257 (609h49m3.305964302s) [noise=true , address=145.79.227.38:4978 , piece_id=KHFQVGJUDBW44VD373WU5TVY53Q2DWWAZWHYEFKJ7WEZTGZZE32A, seg_pos={0 0}, upload=534427]: Done uploading piece. err: <nil> Dec 31 01:18:03.437840822 (609h49m3.308923857s) [noise=true , address=145.79.227.7:4295 , piece_id=4ZTDLM7OGMPW5AL76RLVE5P6NPKSV35YHUUQXG22GILM2KM3HF7Q, seg_pos={0 0}, upload=534427]: Done uploading piece. err: <nil> Dec 31 01:18:03.456780516 (609h49m3.327863551s) [noise=true , address=145.79.227.15:4875 , piece_id=52VZIYF6XEUDVRVPPJLH4KKAKP7NBG6T2K57HEYSWQU7GZ3JQPDQ, seg_pos={0 0}, upload=534427]: Done uploading piece. err: <nil> Dec 31 01:18:03.502042400 (609h49m3.373125445s) [noise=true , address=145.79.227.8:4329 , piece_id=REQU7WRHQIPH5ZYHSXU3LQGXY3ZCZ5AL5KJVS6ZNJWSR3WTSGA7Q, seg_pos={0 0}, upload=534427]: Done uploading piece. err: <nil> Dec 31 01:18:03.507602356 (609h49m3.378685401s) [noise=true , address=145.79.227.25:4619 , piece_id=O4O5UKATE3GDV7XRVPOEQ7ZLHRZD6TCVFRWTCUZIFASKQOCA77GQ, seg_pos={0 0}, upload=534427]: Done uploading piece. err: <nil> Dec 31 01:18:03.757588541 (609h49m3.628671576s) [noise=true , address=45.91.116.9:4262 , piece_id=XF6UPSTRGN2G7UMPNRVTRZB4HSQIPVOJZ6WVZA5PCTG5CHVSADDA, seg_pos={0 0}, upload=534427]: Done uploading piece. err: <nil> Dec 31 01:18:03.826754631 (609h49m3.697837666s) [noise=true , address=45.91.116.5:4299 , piece_id=MDJNFXBJHDNE4WPAZL2P25CKBQH5TNQ4SGLMB7BHJARE4HI26PPA, seg_pos={0 0}, upload=534427]: Done uploading piece. err: <nil> Dec 31 01:18:03.829633754 (609h49m3.700716799s) [noise=true , address=77.90.31.12:4590 , piece_id=WUNBDHFAKDWGSMIKZPARJTDRWLFKLZIBTLB3X6K6OS3SNCP4B5RQ, seg_pos={0 0}, upload=534427]: Done uploading piece. err: <nil> Dec 31 01:18:04.476231966 (609h49m4.347315011s) [noise=true , address=45.91.116.7:4009 , piece_id=TOHJCZG6Q34RDX5ZTZFVJLWVJEF7M27WXQTQORUT3KXSXHQEOVWA, seg_pos={0 0}, upload=534427]: Done uploading piece. err: <nil> Dec 31 01:18:04.476258436 (609h49m4.347341481s) [seg_pos={0 0}, upload=534427]: Segment reached optimal threshold of 13 pieces. Dec 31 01:18:04.476498914 (609h49m4.347581959s) [noise=true , address=45.91.116.14:4189 , piece_id=OWINAXCRNMWKQVZTHGVW7BNQXO7CNSTZBRKC26CU5KHG6ARW5AEQ, seg_pos={0 0}, upload=534427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:40570->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:40570->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40570->45.91.116.14:4189: use of closed network connection Dec 31 01:18:04.476509103 (609h49m4.347592148s) [noise=true , address=45.91.116.14:4189 , piece_id=OWINAXCRNMWKQVZTHGVW7BNQXO7CNSTZBRKC26CU5KHG6ARW5AEQ, seg_pos={0 0}, upload=534427]: Long tail cancellation due to optimal threshold Dec 31 01:18:04.476732279 (609h49m4.347815324s) [noise=true , address=77.90.31.16:4736 , piece_id=J5YEC3SGYIKLGYUSSQH5TNRY7JTBNYUIOJQVJROWHEFISF6ICZWA, seg_pos={0 0}, upload=534427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:53742->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:53742->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53742->77.90.31.16:4736: use of closed network connection Dec 31 01:18:04.476752707 (609h49m4.347835752s) [noise=true , address=77.90.31.16:4736 , piece_id=J5YEC3SGYIKLGYUSSQH5TNRY7JTBNYUIOJQVJROWHEFISF6ICZWA, seg_pos={0 0}, upload=534427]: Long tail cancellation due to optimal threshold Dec 31 01:18:04.476826478 (609h49m4.347909523s) [noise=true , address=77.90.31.19:4844 , piece_id=ZVLY6AIG2ZQQ5F6EUZYLKS7KWRZ3XW665IXSCL55JWBTSREXT42A, seg_pos={0 0}, upload=534427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:60852->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:60852->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60852->77.90.31.19:4844: use of closed network connection Dec 31 01:18:04.476837680 (609h49m4.347920725s) [noise=true , address=77.90.31.19:4844 , piece_id=ZVLY6AIG2ZQQ5F6EUZYLKS7KWRZ3XW665IXSCL55JWBTSREXT42A, seg_pos={0 0}, upload=534427]: Long tail cancellation due to optimal threshold Dec 31 01:18:04.476900229 (609h49m4.347983274s) [seg_pos={0 0}, upload=534427]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:18:04.476999367 (609h49m4.348082412s) [upload=534427]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:18:04.477006361 (609h49m4.348089406s) [upload=534427]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:18:04.775072425 (609h49m4.64615548s) [upload=534427]: Done uploading segments... Dec 31 01:18:04.775097052 (609h49m4.646180097s) [upload=534427]: Upload finished. err: <nil> Duration: 30.002596204s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.508266682s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:18:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.508291909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX.....X...X] duration: 30.003068855s trace: https://jaeger.dev.storj.io/trace/2252fdebffb4f38b FAIL (2025-12-31 01:18:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 31 01:17:30.924871348 (609h48m30.795954383s) [upload=534421]: Starting upload Dec 31 01:17:30.924931543 (609h48m30.796014578s) [upload=534421]: Done starting upload Dec 31 01:17:30.925068544 (609h48m30.796151589s) [upload=534421]: Uploading segments... Dec 31 01:17:30.925157252 (609h48m30.796240287s) [upload=534421]: Got next segment. Inline: false Dec 31 01:17:30.925188822 (609h48m30.796271857s) [upload=534421]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:17:30.925194182 (609h48m30.796277217s) [upload=534421]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:17:31.168262044 (609h48m31.039345079s) [seg_pos={0 0}, upload=534421]: Begin upload segment... Dec 31 01:17:31.168308242 (609h48m31.039391277s) [seg_pos={0 0}, upload=534421]: Done begin upload segment. Dec 31 01:17:31.168338079 (609h48m31.039421114s) [noise=true , address=145.79.227.41:4085 , piece_id=4RNEA6DOWQZR2KVDNU6S4VT5N5QTX6TRC4SOYOZHKDKUUGVWI5BQ, seg_pos={0 0}, upload=534421]: Uploading piece... Dec 31 01:17:31.168420796 (609h48m31.039503831s) [noise=true , address=145.79.227.14:4839 , piece_id=7VRCSGT7GBYW4HXHN2Y4HDWWH5JSYAYQ7AOI3BI6OO7YVV2ANUJA, seg_pos={0 0}, upload=534421]: Uploading piece... Dec 31 01:17:31.169160455 (609h48m31.04024349s) [noise=true , address=145.79.227.25:4618 , piece_id=GQMCIS34MSTHYQJEG4JH3UJOH7CPABVCGDV72OSJLMPNV36RXL4A, seg_pos={0 0}, upload=534421]: Uploading piece... Dec 31 01:17:31.169701695 (609h48m31.04078473s) [noise=true , address=145.79.227.45:4229 , piece_id=6ZARQ2HYXXL5BOT2VCGKJ36Y4IVDMWRFJ644UP6YXMJ2SW35HC4Q, seg_pos={0 0}, upload=534421]: Uploading piece... Dec 31 01:17:31.170071199 (609h48m31.041154234s) [noise=true , address=45.91.116.7:4010 , piece_id=GZXROLAQFSHB6QOM27JKTFDBEEUFWEAX3A6ZAND4ZQCFJC4BBCZQ, seg_pos={0 0}, upload=534421]: Uploading piece... Dec 31 01:17:31.170646895 (609h48m31.04172994s) [noise=true , address=145.79.227.16:4910 , piece_id=G543LU77Q7C4E56TEZCVUJTQ4DLWNRRLUUCQ4CAQGJCVWTSEKJJA, seg_pos={0 0}, upload=534421]: Uploading piece... Dec 31 01:17:31.170814263 (609h48m31.041897309s) [noise=true , address=145.79.227.17:4971 , piece_id=23BWEIHDRLXGV6XKCKWLT2KJT6QRB3P2WKLHF3GW4VJXRWBKQLYQ, seg_pos={0 0}, upload=534421]: Uploading piece... Dec 31 01:17:31.171000137 (609h48m31.042083182s) [noise=true , address=145.79.227.10:4401 , piece_id=T3ABMUZC5ZWHYOFKQBWYUQHS2FDQJ2AQPIJPDC6YHPANDDUVDETQ, seg_pos={0 0}, upload=534421]: Uploading piece... Dec 31 01:17:31.171260604 (609h48m31.042343639s) [noise=true , address=145.79.227.51:4410 , piece_id=U4FP3W436MP66VVB64WHMBT73ZMVQU2SIXZR53ORRHDDNSO2OZ4Q, seg_pos={0 0}, upload=534421]: Uploading piece... Dec 31 01:17:31.171531610 (609h48m31.042614645s) [noise=true , address=45.91.116.9:4261 , piece_id=IQE42J5POAWRPOFTLT4ITZRSMGQYIAM7EBXQJAGTT6OPFEIXZ2UA, seg_pos={0 0}, upload=534421]: Uploading piece... Dec 31 01:17:31.171799569 (609h48m31.042882604s) [noise=true , address=145.79.227.37:4565 , piece_id=OFPZXPMJTV62OVOR7N7B6KKXCJHJJCRALWLHNGMYQEWZNG6W7G7Q, seg_pos={0 0}, upload=534421]: Uploading piece... Dec 31 01:17:31.171911452 (609h48m31.042994497s) [noise=true , address=77.90.31.10:4517 , piece_id=UCYITBNTK4GWCLJ7S7OITCGUEJLXFSPKBD7RIAY64PWYVVLEXWVA, seg_pos={0 0}, upload=534421]: Uploading piece... Dec 31 01:17:31.172314510 (609h48m31.043397545s) [noise=true , address=45.91.116.6:4975 , piece_id=MJHDJTDPYOXOKOKM4JY2UGJE5PCJRXAKQCJWEWCAOFKDXECE5LUQ, seg_pos={0 0}, upload=534421]: Uploading piece... Dec 31 01:17:31.172323136 (609h48m31.043406171s) [noise=true , address=45.91.116.13:4155 , piece_id=CJDPJCOB2DT72K6UZTLUHM7HTH3WHKP6VCJI5KRVIHMB2C5YQ62A, seg_pos={0 0}, upload=534421]: Uploading piece... Dec 31 01:17:31.172758596 (609h48m31.043841631s) [noise=true , address=145.79.227.15:4875 , piece_id=FXV6BKLWWG5BIDC6T743YUYQCVTBKBRQ6IU4WWTHQS37EIGAIMHA, seg_pos={0 0}, upload=534421]: Uploading piece... Dec 31 01:17:31.172896188 (609h48m31.043979223s) [noise=true , address=77.90.31.19:4842 , piece_id=NB53KDFTWL7LLS4LIWRRZ2AYFHVEZDFASK3SF53UW73MFDCHSW6Q, seg_pos={0 0}, upload=534421]: Uploading piece... Dec 31 01:17:32.866019372 (609h48m32.737102417s) [upload=534421]: Next returned nil segment Dec 31 01:17:32.866093934 (609h48m32.737176979s) [upload=534421]: Waiting for error group managing segments... Dec 31 01:17:33.146809150 (609h48m33.017892185s) [noise=true , address=145.79.227.51:4410 , piece_id=U4FP3W436MP66VVB64WHMBT73ZMVQU2SIXZR53ORRHDDNSO2OZ4Q, seg_pos={0 0}, upload=534421]: Done uploading piece. err: <nil> Dec 31 01:17:33.153534676 (609h48m33.024617712s) [noise=true , address=145.79.227.16:4910 , piece_id=G543LU77Q7C4E56TEZCVUJTQ4DLWNRRLUUCQ4CAQGJCVWTSEKJJA, seg_pos={0 0}, upload=534421]: Done uploading piece. err: <nil> Dec 31 01:17:33.174615077 (609h48m33.045698122s) [noise=true , address=145.79.227.37:4565 , piece_id=OFPZXPMJTV62OVOR7N7B6KKXCJHJJCRALWLHNGMYQEWZNG6W7G7Q, seg_pos={0 0}, upload=534421]: Done uploading piece. err: <nil> Dec 31 01:17:33.175488870 (609h48m33.046571915s) [noise=true , address=145.79.227.10:4401 , piece_id=T3ABMUZC5ZWHYOFKQBWYUQHS2FDQJ2AQPIJPDC6YHPANDDUVDETQ, seg_pos={0 0}, upload=534421]: Done uploading piece. err: <nil> Dec 31 01:17:33.180007043 (609h48m33.051090078s) [noise=true , address=145.79.227.45:4229 , piece_id=6ZARQ2HYXXL5BOT2VCGKJ36Y4IVDMWRFJ644UP6YXMJ2SW35HC4Q, seg_pos={0 0}, upload=534421]: Done uploading piece. err: <nil> Dec 31 01:17:33.185262651 (609h48m33.056345686s) [noise=true , address=145.79.227.14:4839 , piece_id=7VRCSGT7GBYW4HXHN2Y4HDWWH5JSYAYQ7AOI3BI6OO7YVV2ANUJA, seg_pos={0 0}, upload=534421]: Done uploading piece. err: <nil> Dec 31 01:17:33.186067483 (609h48m33.057150528s) [noise=true , address=145.79.227.25:4618 , piece_id=GQMCIS34MSTHYQJEG4JH3UJOH7CPABVCGDV72OSJLMPNV36RXL4A, seg_pos={0 0}, upload=534421]: Done uploading piece. err: <nil> Dec 31 01:17:33.228873291 (609h48m33.099956326s) [noise=true , address=145.79.227.17:4971 , piece_id=23BWEIHDRLXGV6XKCKWLT2KJT6QRB3P2WKLHF3GW4VJXRWBKQLYQ, seg_pos={0 0}, upload=534421]: Done uploading piece. err: <nil> Dec 31 01:17:33.240699389 (609h48m33.111782424s) [noise=true , address=145.79.227.41:4085 , piece_id=4RNEA6DOWQZR2KVDNU6S4VT5N5QTX6TRC4SOYOZHKDKUUGVWI5BQ, seg_pos={0 0}, upload=534421]: Done uploading piece. err: <nil> Dec 31 01:17:33.256356803 (609h48m33.127439838s) [noise=true , address=145.79.227.15:4875 , piece_id=FXV6BKLWWG5BIDC6T743YUYQCVTBKBRQ6IU4WWTHQS37EIGAIMHA, seg_pos={0 0}, upload=534421]: Done uploading piece. err: <nil> Dec 31 01:17:33.401383642 (609h48m33.272466677s) [noise=true , address=77.90.31.10:4517 , piece_id=UCYITBNTK4GWCLJ7S7OITCGUEJLXFSPKBD7RIAY64PWYVVLEXWVA, seg_pos={0 0}, upload=534421]: Done uploading piece. err: <nil> Dec 31 01:17:33.446824587 (609h48m33.317907632s) [noise=true , address=45.91.116.6:4975 , piece_id=MJHDJTDPYOXOKOKM4JY2UGJE5PCJRXAKQCJWEWCAOFKDXECE5LUQ, seg_pos={0 0}, upload=534421]: Done uploading piece. err: <nil> Dec 31 01:17:33.468108574 (609h48m33.339191619s) [noise=true , address=45.91.116.13:4155 , piece_id=CJDPJCOB2DT72K6UZTLUHM7HTH3WHKP6VCJI5KRVIHMB2C5YQ62A, seg_pos={0 0}, upload=534421]: Done uploading piece. err: <nil> Dec 31 01:17:33.468127370 (609h48m33.339210405s) [seg_pos={0 0}, upload=534421]: Segment reached optimal threshold of 13 pieces. Dec 31 01:17:33.468643172 (609h48m33.339726218s) [noise=true , address=45.91.116.7:4010 , piece_id=GZXROLAQFSHB6QOM27JKTFDBEEUFWEAX3A6ZAND4ZQCFJC4BBCZQ, seg_pos={0 0}, upload=534421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 31 01:17:33.468652300 (609h48m33.339735335s) [noise=true , address=45.91.116.7:4010 , piece_id=GZXROLAQFSHB6QOM27JKTFDBEEUFWEAX3A6ZAND4ZQCFJC4BBCZQ, seg_pos={0 0}, upload=534421]: Long tail cancellation due to optimal threshold Dec 31 01:17:33.469081787 (609h48m33.340164822s) [noise=true , address=45.91.116.9:4261 , piece_id=IQE42J5POAWRPOFTLT4ITZRSMGQYIAM7EBXQJAGTT6OPFEIXZ2UA, seg_pos={0 0}, upload=534421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 31 01:17:33.469090314 (609h48m33.340173359s) [noise=true , address=45.91.116.9:4261 , piece_id=IQE42J5POAWRPOFTLT4ITZRSMGQYIAM7EBXQJAGTT6OPFEIXZ2UA, seg_pos={0 0}, upload=534421]: Long tail cancellation due to optimal threshold Dec 31 01:17:33.469299673 (609h48m33.340382708s) [noise=true , address=77.90.31.19:4842 , piece_id=NB53KDFTWL7LLS4LIWRRZ2AYFHVEZDFASK3SF53UW73MFDCHSW6Q, seg_pos={0 0}, upload=534421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:47052->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47052->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47052->77.90.31.19:4842: use of closed network connection Dec 31 01:17:33.469308209 (609h48m33.340391254s) [noise=true , address=77.90.31.19:4842 , piece_id=NB53KDFTWL7LLS4LIWRRZ2AYFHVEZDFASK3SF53UW73MFDCHSW6Q, seg_pos={0 0}, upload=534421]: Long tail cancellation due to optimal threshold Dec 31 01:17:33.469414661 (609h48m33.340497706s) [seg_pos={0 0}, upload=534421]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:17:33.469599743 (609h48m33.340682778s) [upload=534421]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:17:33.469605775 (609h48m33.34068881s) [upload=534421]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:17:33.760341271 (609h48m33.631424307s) [upload=534421]: Done uploading segments... Dec 31 01:17:33.760364926 (609h48m33.631447971s) [upload=534421]: Upload finished. err: <nil> Duration: 30.003090516s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.105392424s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:17:44 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.105420919s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.........................................X...........X...] duration: 15.410768626s trace: https://jaeger.dev.storj.io/trace/1a677c844092c76b FAIL (2025-12-31 01:17:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.800832805 (609h45m28.67191585s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801258315 (609h45m28.67234135s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801566793 (609h45m28.672649828s) [noise=true , address=64.49.13.170:4727 , piece_id=4RMUNBIBYA7HZ4Y6XQSSHY4Z6Q26RUA3OBJXKWLZ5X53B5UJHFUA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801962346 (609h45m28.673045391s) [noise=true , address=77.90.31.10:4526 , piece_id=35AWY74SYNHQAXJXRIMWXRENPCA7C2KEM3IZV5H7RK2R3REV4NVA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.802631961 (609h45m28.673715006s) [noise=true , address=145.79.227.40:4059 , piece_id=MH7WUTLHFU7PH5CPP6EZFQEFLVGLG4TZP2PZROUA4A5VAQJWBHIQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.803220893 (609h45m28.674303928s) [noise=true , address=145.79.227.6:4280 , piece_id=DVS4HVIFEX6HY3NPBQC5J2PTTDCQ3Y7LRAHA76FAQKGMMVRK5RLA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804321939 (609h45m28.675404984s) [noise=true , address=45.91.116.7:4027 , piece_id=BAJMVHMEBVKC753XA6ZBO7YVFKR4IOTVGKTJIUSLKFXDTJENRMEQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804615839 (609h45m28.675698874s) [noise=true , address=145.79.227.28:4743 , piece_id=4I4EL63MIAAIUV3WS2SKF6H2EM3II4TL5SEXDZHSD2HQR35RAZUQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804969001 (609h45m28.676052036s) [noise=true , address=77.90.31.12:4617 , piece_id=5HOAYWDAAIPPGAEMZ2RGYD33ADMQK2ARK6VA7G77HTAGP3SEDXZQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.805292477 (609h45m28.676375512s) [noise=true , address=77.90.31.14:4684 , piece_id=UOGNPGQMJ7S5LSQDC7KKKBQ2MEZRXGFGWHLAUC54R6PKRGQ3MDVQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.805617435 (609h45m28.676700471s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.806029400 (609h45m28.677112445s) [noise=true , address=145.79.227.16:4918 , piece_id=KTNWZCL4OHO34WOREGQ5LBMANTRX5NKS6SMPFJUV7JQSTKSTQYDA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.806610056 (609h45m28.677693111s) [noise=true , address=145.79.227.9:4375 , piece_id=DBIJYHETD5TOBQQBQBQUYXWYENNTPCCTTR6NKYWO36XDPDIHJGRQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:30.220457684 (609h45m30.091540719s) [upload=534376]: Next returned nil segment Dec 31 01:14:30.220473895 (609h45m30.09155693s) [upload=534376]: Waiting for error group managing segments... Dec 31 01:14:30.582370671 (609h45m30.453453706s) [noise=true , address=64.49.13.170:4727 , piece_id=4RMUNBIBYA7HZ4Y6XQSSHY4Z6Q26RUA3OBJXKWLZ5X53B5UJHFUA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.660359852 (609h45m30.531442897s) [noise=true , address=145.79.227.29:4961 , piece_id=FL4ZAH3X7BJZZJZ2VLAOXZSS7DZ5NGUYGEYX2II7J6P6ERRKMWSA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.669673326 (609h45m30.540756361s) [noise=true , address=145.79.227.6:4280 , piece_id=DVS4HVIFEX6HY3NPBQC5J2PTTDCQ3Y7LRAHA76FAQKGMMVRK5RLA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.669821237 (609h45m30.540904272s) [noise=true , address=145.79.227.16:4918 , piece_id=KTNWZCL4OHO34WOREGQ5LBMANTRX5NKS6SMPFJUV7JQSTKSTQYDA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.723128962 (609h45m30.594212007s) [noise=true , address=145.79.227.9:4375 , piece_id=DBIJYHETD5TOBQQBQBQUYXWYENNTPCCTTR6NKYWO36XDPDIHJGRQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.836497239 (609h45m30.707580284s) [noise=true , address=77.90.31.14:4684 , piece_id=UOGNPGQMJ7S5LSQDC7KKKBQ2MEZRXGFGWHLAUC54R6PKRGQ3MDVQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.847254552 (609h45m30.718337587s) [noise=true , address=77.90.31.12:4617 , piece_id=5HOAYWDAAIPPGAEMZ2RGYD33ADMQK2ARK6VA7G77HTAGP3SEDXZQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.857173689 (609h45m30.728256734s) [noise=true , address=77.90.31.10:4526 , piece_id=35AWY74SYNHQAXJXRIMWXRENPCA7C2KEM3IZV5H7RK2R3REV4NVA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.956827463 (609h45m30.827910498s) [noise=true , address=45.91.116.7:4027 , piece_id=BAJMVHMEBVKC753XA6ZBO7YVFKR4IOTVGKTJIUSLKFXDTJENRMEQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.987365921 (609h45m30.858448956s) [noise=true , address=145.79.227.28:4743 , piece_id=4I4EL63MIAAIUV3WS2SKF6H2EM3II4TL5SEXDZHSD2HQR35RAZUQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.032233144 (609h45m30.903316179s) [noise=true , address=45.91.116.8:4055 , piece_id=SC3SJFOKLLANT6EZC42UWON7YW73HGZAWICGMV7EY2CGS5VO3R4A, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.549259104 (609h45m31.420342149s) [noise=true , address=145.79.227.40:4059 , piece_id=MH7WUTLHFU7PH5CPP6EZFQEFLVGLG4TZP2PZROUA4A5VAQJWBHIQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.563439376 (609h45m31.434522421s) [noise=true , address=145.79.227.43:4171 , piece_id=NUT3FLC5ORYYHMESA5MRPQSYKSLVBMNDURGSW4LG5MW4WHZZCHWA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.613378095 (609h45m31.48446114s) [noise=true , address=145.79.227.44:4204 , piece_id=MSLSLTQFLQDKLOUDSD3BVSTD54NGEDE72EHWHYZCWT7AX3GZF2VQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.613397552 (609h45m31.484480597s) [seg_pos={0 0}, upload=534376]: Segment reached optimal threshold of 14 pieces. Dec 31 01:14:31.613581703 (609h45m31.484664738s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12238rvPqrDXBowLsUiKTykdZFgfyq4fX9YdaeLa52sQJCkRMHM): protocol: expected piece hash; context canceled; EOF Dec 31 01:14:31.613588355 (609h45m31.48467139s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613657807 (609h45m31.484740853s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta): protocol: write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection; write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection Dec 31 01:14:31.613666333 (609h45m31.484749369s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613690650 (609h45m31.484773695s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL): protocol: write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection; write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection Dec 31 01:14:31.613696842 (609h45m31.484779887s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613847469 (609h45m31.484930514s) [noise=true , address=145.79.227.4:4216 , piece_id=MYGW7QWRK2UF4J4J3LGYZCKFITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgazvjebAo7K4NdQ7zsKkETiNQNfiykHcBDcmXej6eipwWpFu): protocol: write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection; write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection Dec 31 01:14:31.613854532 (609h45m31.484937577s) [noise=true , address=145.79.227.4:4216 , piece_id=MYGW7QWRK2UF4J4J3LGYZCKFITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613947559 (609h45m31.485030594s) [seg_pos={0 0}, upload=534376]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:14:31.614017652 (609h45m31.485100697s) [upload=534376]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:14:31.614023655 (609h45m31.48510669s) [upload=534376]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:14:31.917383268 (609h45m31.788466313s) [upload=534376]: Done uploading segments... Dec 31 01:14:31.917408336 (609h45m31.788491381s) [upload=534376]: Upload finished. err: <nil> Duration: 15.410797831s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.61456654s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:17:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.614595424s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.51715491s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:16:47 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.517184966s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX.....X...] duration: 23.952296968s trace: https://jaeger.dev.storj.io/trace/5c211acadf0a2f7c FAIL (2025-12-31 01:16:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:13:31.892162068 (609h44m31.763245103s) [upload=534365]: Starting upload Dec 31 01:13:31.892229988 (609h44m31.763313023s) [upload=534365]: Done starting upload Dec 31 01:13:31.892520051 (609h44m31.763603086s) [upload=534365]: Uploading segments... Dec 31 01:13:31.892835572 (609h44m31.763918607s) [upload=534365]: Got next segment. Inline: false Dec 31 01:13:31.892880577 (609h44m31.763963622s) [upload=534365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:13:31.892886969 (609h44m31.763970014s) [upload=534365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:13:32.136582035 (609h44m32.00766508s) [seg_pos={0 0}, upload=534365]: Begin upload segment... Dec 31 01:13:32.136741648 (609h44m32.007824693s) [seg_pos={0 0}, upload=534365]: Done begin upload segment. Dec 31 01:13:32.136787886 (609h44m32.007870921s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.136952009 (609h44m32.008035054s) [noise=true , address=145.79.227.7:4293 , piece_id=PAWHOP4S5N2SWCW2NVTGSGGGZDYKVZ52HE5LUXWBC45WTCZ47LXQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137392508 (609h44m32.008475553s) [noise=true , address=145.79.227.20:4438 , piece_id=HHBPYPKJUJOGJA76IYPO3QBCURQBXSZAMDRI6UGEV4RAGKSAZ3FQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137833196 (609h44m32.008916241s) [noise=true , address=145.79.227.22:4511 , piece_id=UEOO6Y5AFCQ5RPEFKQHNXIM6DQ6CNVGJ46SU7MLECWVP6NEGV4WQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137887390 (609h44m32.008970435s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138202520 (609h44m32.009285555s) [noise=true , address=45.91.116.7:4011 , piece_id=4SKDVZH2AHRVLV5362AOOCGLOBIPEK7WGDBMWLGWZB3Q6NNLQPSA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138443369 (609h44m32.009526414s) [noise=true , address=145.79.227.37:4565 , piece_id=YYTOH6DR725IZRPAGNOOTZCSQEI55FEYKDL4PNF3SYONLMOHQ25A, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138677505 (609h44m32.00976054s) [noise=true , address=145.79.227.39:4014 , piece_id=ZB3AA4NG3QF6PLHH6GTGD7PF7FAXUBOEPWVOIOJ5JRQLAZOVBGQA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138997634 (609h44m32.010080679s) [noise=true , address=145.79.227.5:4222 , piece_id=AU6TDEXDLGFCXPSFIKYQ3AM4UXBXVMXZ2RVIILMBX6I6UZWU7CJQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139179520 (609h44m32.010262565s) [noise=true , address=145.79.227.3:4150 , piece_id=LRCVVPU4EXBCU6T4FF4PJUUSJYM4RCR5WPVGAYCAWBFIOE77QQRQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139293667 (609h44m32.010376712s) [noise=true , address=45.91.116.6:4973 , piece_id=IUHID3DDNNPTIAQXX4UCETB5LFFJBYHM7POMA7ALIZYZALICQ73Q, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139647271 (609h44m32.010730316s) [noise=true , address=77.90.31.16:4735 , piece_id=DDIONJAYZBASQJDP3QPISHXACAFZGOW3EBSMIXNNT2LY4DLYXADQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139782598 (609h44m32.010865643s) [noise=true , address=45.91.116.14:4189 , piece_id=FAPD7HNXB3TV33NFDF4UFJGHY7AWSPTL3KZHMQ4QQIYZ7ZEMSJGA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.140427918 (609h44m32.011510953s) [noise=true , address=145.79.227.2:4113 , piece_id=A2U3D2C7LEPHVPU3IUKBNNM6BFHBW2AOMRRIZ6XPB2MEIANKPRXQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.141112661 (609h44m32.012195706s) [noise=true , address=77.90.31.21:4877 , piece_id=BBOICMOZ3RSNP3I5XX2AQW3B6SURUFUDCTZG25KXP62FSNOSRQUQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.141783057 (609h44m32.012866102s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:33.837053250 (609h44m33.708136285s) [upload=534365]: Next returned nil segment Dec 31 01:13:33.837088717 (609h44m33.708171752s) [upload=534365]: Waiting for error group managing segments... Dec 31 01:13:34.123562999 (609h44m33.994646034s) [noise=true , address=145.79.227.7:4293 , piece_id=PAWHOP4S5N2SWCW2NVTGSGGGZDYKVZ52HE5LUXWBC45WTCZ47LXQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.124260658 (609h44m33.995343693s) [noise=true , address=145.79.227.20:4438 , piece_id=HHBPYPKJUJOGJA76IYPO3QBCURQBXSZAMDRI6UGEV4RAGKSAZ3FQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.125012440 (609h44m33.996095485s) [noise=true , address=145.79.227.22:4511 , piece_id=UEOO6Y5AFCQ5RPEFKQHNXIM6DQ6CNVGJ46SU7MLECWVP6NEGV4WQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.132053355 (609h44m34.003136411s) [noise=true , address=145.79.227.39:4014 , piece_id=ZB3AA4NG3QF6PLHH6GTGD7PF7FAXUBOEPWVOIOJ5JRQLAZOVBGQA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.135253870 (609h44m34.006336905s) [noise=true , address=145.79.227.3:4150 , piece_id=LRCVVPU4EXBCU6T4FF4PJUUSJYM4RCR5WPVGAYCAWBFIOE77QQRQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.138185082 (609h44m34.009268117s) [noise=true , address=145.79.227.5:4222 , piece_id=AU6TDEXDLGFCXPSFIKYQ3AM4UXBXVMXZ2RVIILMBX6I6UZWU7CJQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.145248080 (609h44m34.016331116s) [noise=true , address=145.79.227.37:4565 , piece_id=YYTOH6DR725IZRPAGNOOTZCSQEI55FEYKDL4PNF3SYONLMOHQ25A, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.146704714 (609h44m34.017787779s) [noise=true , address=145.79.227.2:4113 , piece_id=A2U3D2C7LEPHVPU3IUKBNNM6BFHBW2AOMRRIZ6XPB2MEIANKPRXQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.312085941 (609h44m34.183168976s) [noise=true , address=77.90.31.16:4735 , piece_id=DDIONJAYZBASQJDP3QPISHXACAFZGOW3EBSMIXNNT2LY4DLYXADQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.436213054 (609h44m34.307296109s) [noise=true , address=45.91.116.14:4189 , piece_id=FAPD7HNXB3TV33NFDF4UFJGHY7AWSPTL3KZHMQ4QQIYZ7ZEMSJGA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.766896134 (609h44m34.637979179s) [noise=true , address=45.91.116.7:4011 , piece_id=4SKDVZH2AHRVLV5362AOOCGLOBIPEK7WGDBMWLGWZB3Q6NNLQPSA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.034120011 (609h44m34.905203046s) [noise=true , address=77.90.31.21:4877 , piece_id=BBOICMOZ3RSNP3I5XX2AQW3B6SURUFUDCTZG25KXP62FSNOSRQUQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.317152980 (609h44m35.188236025s) [noise=true , address=45.91.116.6:4973 , piece_id=IUHID3DDNNPTIAQXX4UCETB5LFFJBYHM7POMA7ALIZYZALICQ73Q, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.317171395 (609h44m35.18825443s) [seg_pos={0 0}, upload=534365]: Segment reached optimal threshold of 13 pieces. Dec 31 01:13:35.317363992 (609h44m35.188447027s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317369673 (609h44m35.188452708s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317497957 (609h44m35.188580992s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317503487 (609h44m35.188586522s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317612284 (609h44m35.188695339s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317643834 (609h44m35.188726869s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317682628 (609h44m35.188765663s) [seg_pos={0 0}, upload=534365]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:13:35.317778912 (609h44m35.188861947s) [upload=534365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:13:35.317784572 (609h44m35.188867607s) [upload=534365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:13:35.615134704 (609h44m35.486217749s) [upload=534365]: Done uploading segments... Dec 31 01:13:35.615159452 (609h44m35.486242497s) [upload=534365]: Upload finished. err: <nil> Duration: 23.952325311s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.........................................X...........X..] duration: 17.437483064s trace: https://jaeger.dev.storj.io/trace/1a677c844092c76b FAIL (2025-12-31 01:16:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.800832805 (609h45m28.67191585s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801258315 (609h45m28.67234135s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801566793 (609h45m28.672649828s) [noise=true , address=64.49.13.170:4727 , piece_id=4RMUNBIBYA7HZ4Y6XQSSHY4Z6Q26RUA3OBJXKWLZ5X53B5UJHFUA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801962346 (609h45m28.673045391s) [noise=true , address=77.90.31.10:4526 , piece_id=35AWY74SYNHQAXJXRIMWXRENPCA7C2KEM3IZV5H7RK2R3REV4NVA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.802631961 (609h45m28.673715006s) [noise=true , address=145.79.227.40:4059 , piece_id=MH7WUTLHFU7PH5CPP6EZFQEFLVGLG4TZP2PZROUA4A5VAQJWBHIQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.803220893 (609h45m28.674303928s) [noise=true , address=145.79.227.6:4280 , piece_id=DVS4HVIFEX6HY3NPBQC5J2PTTDCQ3Y7LRAHA76FAQKGMMVRK5RLA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804321939 (609h45m28.675404984s) [noise=true , address=45.91.116.7:4027 , piece_id=BAJMVHMEBVKC753XA6ZBO7YVFKR4IOTVGKTJIUSLKFXDTJENRMEQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804615839 (609h45m28.675698874s) [noise=true , address=145.79.227.28:4743 , piece_id=4I4EL63MIAAIUV3WS2SKF6H2EM3II4TL5SEXDZHSD2HQR35RAZUQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804969001 (609h45m28.676052036s) [noise=true , address=77.90.31.12:4617 , piece_id=5HOAYWDAAIPPGAEMZ2RGYD33ADMQK2ARK6VA7G77HTAGP3SEDXZQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.805292477 (609h45m28.676375512s) [noise=true , address=77.90.31.14:4684 , piece_id=UOGNPGQMJ7S5LSQDC7KKKBQ2MEZRXGFGWHLAUC54R6PKRGQ3MDVQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.805617435 (609h45m28.676700471s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.806029400 (609h45m28.677112445s) [noise=true , address=145.79.227.16:4918 , piece_id=KTNWZCL4OHO34WOREGQ5LBMANTRX5NKS6SMPFJUV7JQSTKSTQYDA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.806610056 (609h45m28.677693111s) [noise=true , address=145.79.227.9:4375 , piece_id=DBIJYHETD5TOBQQBQBQUYXWYENNTPCCTTR6NKYWO36XDPDIHJGRQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:30.220457684 (609h45m30.091540719s) [upload=534376]: Next returned nil segment Dec 31 01:14:30.220473895 (609h45m30.09155693s) [upload=534376]: Waiting for error group managing segments... Dec 31 01:14:30.582370671 (609h45m30.453453706s) [noise=true , address=64.49.13.170:4727 , piece_id=4RMUNBIBYA7HZ4Y6XQSSHY4Z6Q26RUA3OBJXKWLZ5X53B5UJHFUA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.660359852 (609h45m30.531442897s) [noise=true , address=145.79.227.29:4961 , piece_id=FL4ZAH3X7BJZZJZ2VLAOXZSS7DZ5NGUYGEYX2II7J6P6ERRKMWSA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.669673326 (609h45m30.540756361s) [noise=true , address=145.79.227.6:4280 , piece_id=DVS4HVIFEX6HY3NPBQC5J2PTTDCQ3Y7LRAHA76FAQKGMMVRK5RLA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.669821237 (609h45m30.540904272s) [noise=true , address=145.79.227.16:4918 , piece_id=KTNWZCL4OHO34WOREGQ5LBMANTRX5NKS6SMPFJUV7JQSTKSTQYDA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.723128962 (609h45m30.594212007s) [noise=true , address=145.79.227.9:4375 , piece_id=DBIJYHETD5TOBQQBQBQUYXWYENNTPCCTTR6NKYWO36XDPDIHJGRQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.836497239 (609h45m30.707580284s) [noise=true , address=77.90.31.14:4684 , piece_id=UOGNPGQMJ7S5LSQDC7KKKBQ2MEZRXGFGWHLAUC54R6PKRGQ3MDVQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.847254552 (609h45m30.718337587s) [noise=true , address=77.90.31.12:4617 , piece_id=5HOAYWDAAIPPGAEMZ2RGYD33ADMQK2ARK6VA7G77HTAGP3SEDXZQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.857173689 (609h45m30.728256734s) [noise=true , address=77.90.31.10:4526 , piece_id=35AWY74SYNHQAXJXRIMWXRENPCA7C2KEM3IZV5H7RK2R3REV4NVA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.956827463 (609h45m30.827910498s) [noise=true , address=45.91.116.7:4027 , piece_id=BAJMVHMEBVKC753XA6ZBO7YVFKR4IOTVGKTJIUSLKFXDTJENRMEQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.987365921 (609h45m30.858448956s) [noise=true , address=145.79.227.28:4743 , piece_id=4I4EL63MIAAIUV3WS2SKF6H2EM3II4TL5SEXDZHSD2HQR35RAZUQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.032233144 (609h45m30.903316179s) [noise=true , address=45.91.116.8:4055 , piece_id=SC3SJFOKLLANT6EZC42UWON7YW73HGZAWICGMV7EY2CGS5VO3R4A, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.549259104 (609h45m31.420342149s) [noise=true , address=145.79.227.40:4059 , piece_id=MH7WUTLHFU7PH5CPP6EZFQEFLVGLG4TZP2PZROUA4A5VAQJWBHIQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.563439376 (609h45m31.434522421s) [noise=true , address=145.79.227.43:4171 , piece_id=NUT3FLC5ORYYHMESA5MRPQSYKSLVBMNDURGSW4LG5MW4WHZZCHWA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.613378095 (609h45m31.48446114s) [noise=true , address=145.79.227.44:4204 , piece_id=MSLSLTQFLQDKLOUDSD3BVSTD54NGEDE72EHWHYZCWT7AX3GZF2VQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.613397552 (609h45m31.484480597s) [seg_pos={0 0}, upload=534376]: Segment reached optimal threshold of 14 pieces. Dec 31 01:14:31.613581703 (609h45m31.484664738s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12238rvPqrDXBowLsUiKTykdZFgfyq4fX9YdaeLa52sQJCkRMHM): protocol: expected piece hash; context canceled; EOF Dec 31 01:14:31.613588355 (609h45m31.48467139s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613657807 (609h45m31.484740853s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta): protocol: write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection; write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection Dec 31 01:14:31.613666333 (609h45m31.484749369s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613690650 (609h45m31.484773695s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL): protocol: write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection; write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection Dec 31 01:14:31.613696842 (609h45m31.484779887s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613847469 (609h45m31.484930514s) [noise=true , address=145.79.227.4:4216 , piece_id=MYGW7QWRK2UF4J4J3LGYZCKFITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgazvjebAo7K4NdQ7zsKkETiNQNfiykHcBDcmXej6eipwWpFu): protocol: write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection; write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection Dec 31 01:14:31.613854532 (609h45m31.484937577s) [noise=true , address=145.79.227.4:4216 , piece_id=MYGW7QWRK2UF4J4J3LGYZCKFITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613947559 (609h45m31.485030594s) [seg_pos={0 0}, upload=534376]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:14:31.614017652 (609h45m31.485100697s) [upload=534376]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:14:31.614023655 (609h45m31.48510669s) [upload=534376]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:14:31.917383268 (609h45m31.788466313s) [upload=534376]: Done uploading segments... Dec 31 01:14:31.917408336 (609h45m31.788491381s) [upload=534376]: Upload finished. err: <nil> Duration: 17.437511157s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.504170178s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:16:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.504196378s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.309370451s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:16:07 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.309403764s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.40283603s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:15:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.402915681s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.........................................X...........X.] duration: 17.548250581s trace: https://jaeger.dev.storj.io/trace/1a677c844092c76b FAIL (2025-12-31 01:15:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.800832805 (609h45m28.67191585s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801258315 (609h45m28.67234135s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801566793 (609h45m28.672649828s) [noise=true , address=64.49.13.170:4727 , piece_id=4RMUNBIBYA7HZ4Y6XQSSHY4Z6Q26RUA3OBJXKWLZ5X53B5UJHFUA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801962346 (609h45m28.673045391s) [noise=true , address=77.90.31.10:4526 , piece_id=35AWY74SYNHQAXJXRIMWXRENPCA7C2KEM3IZV5H7RK2R3REV4NVA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.802631961 (609h45m28.673715006s) [noise=true , address=145.79.227.40:4059 , piece_id=MH7WUTLHFU7PH5CPP6EZFQEFLVGLG4TZP2PZROUA4A5VAQJWBHIQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.803220893 (609h45m28.674303928s) [noise=true , address=145.79.227.6:4280 , piece_id=DVS4HVIFEX6HY3NPBQC5J2PTTDCQ3Y7LRAHA76FAQKGMMVRK5RLA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804321939 (609h45m28.675404984s) [noise=true , address=45.91.116.7:4027 , piece_id=BAJMVHMEBVKC753XA6ZBO7YVFKR4IOTVGKTJIUSLKFXDTJENRMEQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804615839 (609h45m28.675698874s) [noise=true , address=145.79.227.28:4743 , piece_id=4I4EL63MIAAIUV3WS2SKF6H2EM3II4TL5SEXDZHSD2HQR35RAZUQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804969001 (609h45m28.676052036s) [noise=true , address=77.90.31.12:4617 , piece_id=5HOAYWDAAIPPGAEMZ2RGYD33ADMQK2ARK6VA7G77HTAGP3SEDXZQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.805292477 (609h45m28.676375512s) [noise=true , address=77.90.31.14:4684 , piece_id=UOGNPGQMJ7S5LSQDC7KKKBQ2MEZRXGFGWHLAUC54R6PKRGQ3MDVQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.805617435 (609h45m28.676700471s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.806029400 (609h45m28.677112445s) [noise=true , address=145.79.227.16:4918 , piece_id=KTNWZCL4OHO34WOREGQ5LBMANTRX5NKS6SMPFJUV7JQSTKSTQYDA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.806610056 (609h45m28.677693111s) [noise=true , address=145.79.227.9:4375 , piece_id=DBIJYHETD5TOBQQBQBQUYXWYENNTPCCTTR6NKYWO36XDPDIHJGRQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:30.220457684 (609h45m30.091540719s) [upload=534376]: Next returned nil segment Dec 31 01:14:30.220473895 (609h45m30.09155693s) [upload=534376]: Waiting for error group managing segments... Dec 31 01:14:30.582370671 (609h45m30.453453706s) [noise=true , address=64.49.13.170:4727 , piece_id=4RMUNBIBYA7HZ4Y6XQSSHY4Z6Q26RUA3OBJXKWLZ5X53B5UJHFUA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.660359852 (609h45m30.531442897s) [noise=true , address=145.79.227.29:4961 , piece_id=FL4ZAH3X7BJZZJZ2VLAOXZSS7DZ5NGUYGEYX2II7J6P6ERRKMWSA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.669673326 (609h45m30.540756361s) [noise=true , address=145.79.227.6:4280 , piece_id=DVS4HVIFEX6HY3NPBQC5J2PTTDCQ3Y7LRAHA76FAQKGMMVRK5RLA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.669821237 (609h45m30.540904272s) [noise=true , address=145.79.227.16:4918 , piece_id=KTNWZCL4OHO34WOREGQ5LBMANTRX5NKS6SMPFJUV7JQSTKSTQYDA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.723128962 (609h45m30.594212007s) [noise=true , address=145.79.227.9:4375 , piece_id=DBIJYHETD5TOBQQBQBQUYXWYENNTPCCTTR6NKYWO36XDPDIHJGRQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.836497239 (609h45m30.707580284s) [noise=true , address=77.90.31.14:4684 , piece_id=UOGNPGQMJ7S5LSQDC7KKKBQ2MEZRXGFGWHLAUC54R6PKRGQ3MDVQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.847254552 (609h45m30.718337587s) [noise=true , address=77.90.31.12:4617 , piece_id=5HOAYWDAAIPPGAEMZ2RGYD33ADMQK2ARK6VA7G77HTAGP3SEDXZQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.857173689 (609h45m30.728256734s) [noise=true , address=77.90.31.10:4526 , piece_id=35AWY74SYNHQAXJXRIMWXRENPCA7C2KEM3IZV5H7RK2R3REV4NVA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.956827463 (609h45m30.827910498s) [noise=true , address=45.91.116.7:4027 , piece_id=BAJMVHMEBVKC753XA6ZBO7YVFKR4IOTVGKTJIUSLKFXDTJENRMEQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.987365921 (609h45m30.858448956s) [noise=true , address=145.79.227.28:4743 , piece_id=4I4EL63MIAAIUV3WS2SKF6H2EM3II4TL5SEXDZHSD2HQR35RAZUQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.032233144 (609h45m30.903316179s) [noise=true , address=45.91.116.8:4055 , piece_id=SC3SJFOKLLANT6EZC42UWON7YW73HGZAWICGMV7EY2CGS5VO3R4A, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.549259104 (609h45m31.420342149s) [noise=true , address=145.79.227.40:4059 , piece_id=MH7WUTLHFU7PH5CPP6EZFQEFLVGLG4TZP2PZROUA4A5VAQJWBHIQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.563439376 (609h45m31.434522421s) [noise=true , address=145.79.227.43:4171 , piece_id=NUT3FLC5ORYYHMESA5MRPQSYKSLVBMNDURGSW4LG5MW4WHZZCHWA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.613378095 (609h45m31.48446114s) [noise=true , address=145.79.227.44:4204 , piece_id=MSLSLTQFLQDKLOUDSD3BVSTD54NGEDE72EHWHYZCWT7AX3GZF2VQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.613397552 (609h45m31.484480597s) [seg_pos={0 0}, upload=534376]: Segment reached optimal threshold of 14 pieces. Dec 31 01:14:31.613581703 (609h45m31.484664738s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12238rvPqrDXBowLsUiKTykdZFgfyq4fX9YdaeLa52sQJCkRMHM): protocol: expected piece hash; context canceled; EOF Dec 31 01:14:31.613588355 (609h45m31.48467139s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613657807 (609h45m31.484740853s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta): protocol: write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection; write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection Dec 31 01:14:31.613666333 (609h45m31.484749369s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613690650 (609h45m31.484773695s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL): protocol: write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection; write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection Dec 31 01:14:31.613696842 (609h45m31.484779887s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613847469 (609h45m31.484930514s) [noise=true , address=145.79.227.4:4216 , piece_id=MYGW7QWRK2UF4J4J3LGYZCKFITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgazvjebAo7K4NdQ7zsKkETiNQNfiykHcBDcmXej6eipwWpFu): protocol: write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection; write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection Dec 31 01:14:31.613854532 (609h45m31.484937577s) [noise=true , address=145.79.227.4:4216 , piece_id=MYGW7QWRK2UF4J4J3LGYZCKFITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613947559 (609h45m31.485030594s) [seg_pos={0 0}, upload=534376]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:14:31.614017652 (609h45m31.485100697s) [upload=534376]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:14:31.614023655 (609h45m31.48510669s) [upload=534376]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:14:31.917383268 (609h45m31.788466313s) [upload=534376]: Done uploading segments... Dec 31 01:14:31.917408336 (609h45m31.788491381s) [upload=534376]: Upload finished. err: <nil> Duration: 17.548263527s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.574912883s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:15:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.574958029s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.........................................X...........X] duration: 1m0.003237406s trace: https://jaeger.dev.storj.io/trace/1a677c844092c76b FAIL (2025-12-31 01:15:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.800832805 (609h45m28.67191585s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801258315 (609h45m28.67234135s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801566793 (609h45m28.672649828s) [noise=true , address=64.49.13.170:4727 , piece_id=4RMUNBIBYA7HZ4Y6XQSSHY4Z6Q26RUA3OBJXKWLZ5X53B5UJHFUA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801962346 (609h45m28.673045391s) [noise=true , address=77.90.31.10:4526 , piece_id=35AWY74SYNHQAXJXRIMWXRENPCA7C2KEM3IZV5H7RK2R3REV4NVA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.802631961 (609h45m28.673715006s) [noise=true , address=145.79.227.40:4059 , piece_id=MH7WUTLHFU7PH5CPP6EZFQEFLVGLG4TZP2PZROUA4A5VAQJWBHIQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.803220893 (609h45m28.674303928s) [noise=true , address=145.79.227.6:4280 , piece_id=DVS4HVIFEX6HY3NPBQC5J2PTTDCQ3Y7LRAHA76FAQKGMMVRK5RLA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804321939 (609h45m28.675404984s) [noise=true , address=45.91.116.7:4027 , piece_id=BAJMVHMEBVKC753XA6ZBO7YVFKR4IOTVGKTJIUSLKFXDTJENRMEQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804615839 (609h45m28.675698874s) [noise=true , address=145.79.227.28:4743 , piece_id=4I4EL63MIAAIUV3WS2SKF6H2EM3II4TL5SEXDZHSD2HQR35RAZUQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804969001 (609h45m28.676052036s) [noise=true , address=77.90.31.12:4617 , piece_id=5HOAYWDAAIPPGAEMZ2RGYD33ADMQK2ARK6VA7G77HTAGP3SEDXZQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.805292477 (609h45m28.676375512s) [noise=true , address=77.90.31.14:4684 , piece_id=UOGNPGQMJ7S5LSQDC7KKKBQ2MEZRXGFGWHLAUC54R6PKRGQ3MDVQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.805617435 (609h45m28.676700471s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.806029400 (609h45m28.677112445s) [noise=true , address=145.79.227.16:4918 , piece_id=KTNWZCL4OHO34WOREGQ5LBMANTRX5NKS6SMPFJUV7JQSTKSTQYDA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.806610056 (609h45m28.677693111s) [noise=true , address=145.79.227.9:4375 , piece_id=DBIJYHETD5TOBQQBQBQUYXWYENNTPCCTTR6NKYWO36XDPDIHJGRQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:30.220457684 (609h45m30.091540719s) [upload=534376]: Next returned nil segment Dec 31 01:14:30.220473895 (609h45m30.09155693s) [upload=534376]: Waiting for error group managing segments... Dec 31 01:14:30.582370671 (609h45m30.453453706s) [noise=true , address=64.49.13.170:4727 , piece_id=4RMUNBIBYA7HZ4Y6XQSSHY4Z6Q26RUA3OBJXKWLZ5X53B5UJHFUA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.660359852 (609h45m30.531442897s) [noise=true , address=145.79.227.29:4961 , piece_id=FL4ZAH3X7BJZZJZ2VLAOXZSS7DZ5NGUYGEYX2II7J6P6ERRKMWSA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.669673326 (609h45m30.540756361s) [noise=true , address=145.79.227.6:4280 , piece_id=DVS4HVIFEX6HY3NPBQC5J2PTTDCQ3Y7LRAHA76FAQKGMMVRK5RLA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.669821237 (609h45m30.540904272s) [noise=true , address=145.79.227.16:4918 , piece_id=KTNWZCL4OHO34WOREGQ5LBMANTRX5NKS6SMPFJUV7JQSTKSTQYDA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.723128962 (609h45m30.594212007s) [noise=true , address=145.79.227.9:4375 , piece_id=DBIJYHETD5TOBQQBQBQUYXWYENNTPCCTTR6NKYWO36XDPDIHJGRQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.836497239 (609h45m30.707580284s) [noise=true , address=77.90.31.14:4684 , piece_id=UOGNPGQMJ7S5LSQDC7KKKBQ2MEZRXGFGWHLAUC54R6PKRGQ3MDVQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.847254552 (609h45m30.718337587s) [noise=true , address=77.90.31.12:4617 , piece_id=5HOAYWDAAIPPGAEMZ2RGYD33ADMQK2ARK6VA7G77HTAGP3SEDXZQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.857173689 (609h45m30.728256734s) [noise=true , address=77.90.31.10:4526 , piece_id=35AWY74SYNHQAXJXRIMWXRENPCA7C2KEM3IZV5H7RK2R3REV4NVA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.956827463 (609h45m30.827910498s) [noise=true , address=45.91.116.7:4027 , piece_id=BAJMVHMEBVKC753XA6ZBO7YVFKR4IOTVGKTJIUSLKFXDTJENRMEQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.987365921 (609h45m30.858448956s) [noise=true , address=145.79.227.28:4743 , piece_id=4I4EL63MIAAIUV3WS2SKF6H2EM3II4TL5SEXDZHSD2HQR35RAZUQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.032233144 (609h45m30.903316179s) [noise=true , address=45.91.116.8:4055 , piece_id=SC3SJFOKLLANT6EZC42UWON7YW73HGZAWICGMV7EY2CGS5VO3R4A, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.549259104 (609h45m31.420342149s) [noise=true , address=145.79.227.40:4059 , piece_id=MH7WUTLHFU7PH5CPP6EZFQEFLVGLG4TZP2PZROUA4A5VAQJWBHIQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.563439376 (609h45m31.434522421s) [noise=true , address=145.79.227.43:4171 , piece_id=NUT3FLC5ORYYHMESA5MRPQSYKSLVBMNDURGSW4LG5MW4WHZZCHWA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.613378095 (609h45m31.48446114s) [noise=true , address=145.79.227.44:4204 , piece_id=MSLSLTQFLQDKLOUDSD3BVSTD54NGEDE72EHWHYZCWT7AX3GZF2VQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.613397552 (609h45m31.484480597s) [seg_pos={0 0}, upload=534376]: Segment reached optimal threshold of 14 pieces. Dec 31 01:14:31.613581703 (609h45m31.484664738s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12238rvPqrDXBowLsUiKTykdZFgfyq4fX9YdaeLa52sQJCkRMHM): protocol: expected piece hash; context canceled; EOF Dec 31 01:14:31.613588355 (609h45m31.48467139s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613657807 (609h45m31.484740853s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta): protocol: write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection; write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection Dec 31 01:14:31.613666333 (609h45m31.484749369s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613690650 (609h45m31.484773695s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL): protocol: write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection; write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection Dec 31 01:14:31.613696842 (609h45m31.484779887s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613847469 (609h45m31.484930514s) [noise=true , address=145.79.227.4:4216 , piece_id=MYGW7QWRK2UF4J4J3LGYZCKFITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgazvjebAo7K4NdQ7zsKkETiNQNfiykHcBDcmXej6eipwWpFu): protocol: write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection; write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection Dec 31 01:14:31.613854532 (609h45m31.484937577s) [noise=true , address=145.79.227.4:4216 , piece_id=MYGW7QWRK2UF4J4J3LGYZCKFITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613947559 (609h45m31.485030594s) [seg_pos={0 0}, upload=534376]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:14:31.614017652 (609h45m31.485100697s) [upload=534376]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:14:31.614023655 (609h45m31.48510669s) [upload=534376]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:14:31.917383268 (609h45m31.788466313s) [upload=534376]: Done uploading segments... Dec 31 01:14:31.917408336 (609h45m31.788491381s) [upload=534376]: Upload finished. err: <nil> Duration: 1m0.003251011s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX.....X..] duration: 5.81946758s trace: https://jaeger.dev.storj.io/trace/5c211acadf0a2f7c FAIL (2025-12-31 01:15:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:13:31.892162068 (609h44m31.763245103s) [upload=534365]: Starting upload Dec 31 01:13:31.892229988 (609h44m31.763313023s) [upload=534365]: Done starting upload Dec 31 01:13:31.892520051 (609h44m31.763603086s) [upload=534365]: Uploading segments... Dec 31 01:13:31.892835572 (609h44m31.763918607s) [upload=534365]: Got next segment. Inline: false Dec 31 01:13:31.892880577 (609h44m31.763963622s) [upload=534365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:13:31.892886969 (609h44m31.763970014s) [upload=534365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:13:32.136582035 (609h44m32.00766508s) [seg_pos={0 0}, upload=534365]: Begin upload segment... Dec 31 01:13:32.136741648 (609h44m32.007824693s) [seg_pos={0 0}, upload=534365]: Done begin upload segment. Dec 31 01:13:32.136787886 (609h44m32.007870921s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.136952009 (609h44m32.008035054s) [noise=true , address=145.79.227.7:4293 , piece_id=PAWHOP4S5N2SWCW2NVTGSGGGZDYKVZ52HE5LUXWBC45WTCZ47LXQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137392508 (609h44m32.008475553s) [noise=true , address=145.79.227.20:4438 , piece_id=HHBPYPKJUJOGJA76IYPO3QBCURQBXSZAMDRI6UGEV4RAGKSAZ3FQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137833196 (609h44m32.008916241s) [noise=true , address=145.79.227.22:4511 , piece_id=UEOO6Y5AFCQ5RPEFKQHNXIM6DQ6CNVGJ46SU7MLECWVP6NEGV4WQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137887390 (609h44m32.008970435s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138202520 (609h44m32.009285555s) [noise=true , address=45.91.116.7:4011 , piece_id=4SKDVZH2AHRVLV5362AOOCGLOBIPEK7WGDBMWLGWZB3Q6NNLQPSA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138443369 (609h44m32.009526414s) [noise=true , address=145.79.227.37:4565 , piece_id=YYTOH6DR725IZRPAGNOOTZCSQEI55FEYKDL4PNF3SYONLMOHQ25A, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138677505 (609h44m32.00976054s) [noise=true , address=145.79.227.39:4014 , piece_id=ZB3AA4NG3QF6PLHH6GTGD7PF7FAXUBOEPWVOIOJ5JRQLAZOVBGQA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138997634 (609h44m32.010080679s) [noise=true , address=145.79.227.5:4222 , piece_id=AU6TDEXDLGFCXPSFIKYQ3AM4UXBXVMXZ2RVIILMBX6I6UZWU7CJQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139179520 (609h44m32.010262565s) [noise=true , address=145.79.227.3:4150 , piece_id=LRCVVPU4EXBCU6T4FF4PJUUSJYM4RCR5WPVGAYCAWBFIOE77QQRQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139293667 (609h44m32.010376712s) [noise=true , address=45.91.116.6:4973 , piece_id=IUHID3DDNNPTIAQXX4UCETB5LFFJBYHM7POMA7ALIZYZALICQ73Q, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139647271 (609h44m32.010730316s) [noise=true , address=77.90.31.16:4735 , piece_id=DDIONJAYZBASQJDP3QPISHXACAFZGOW3EBSMIXNNT2LY4DLYXADQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139782598 (609h44m32.010865643s) [noise=true , address=45.91.116.14:4189 , piece_id=FAPD7HNXB3TV33NFDF4UFJGHY7AWSPTL3KZHMQ4QQIYZ7ZEMSJGA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.140427918 (609h44m32.011510953s) [noise=true , address=145.79.227.2:4113 , piece_id=A2U3D2C7LEPHVPU3IUKBNNM6BFHBW2AOMRRIZ6XPB2MEIANKPRXQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.141112661 (609h44m32.012195706s) [noise=true , address=77.90.31.21:4877 , piece_id=BBOICMOZ3RSNP3I5XX2AQW3B6SURUFUDCTZG25KXP62FSNOSRQUQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.141783057 (609h44m32.012866102s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:33.837053250 (609h44m33.708136285s) [upload=534365]: Next returned nil segment Dec 31 01:13:33.837088717 (609h44m33.708171752s) [upload=534365]: Waiting for error group managing segments... Dec 31 01:13:34.123562999 (609h44m33.994646034s) [noise=true , address=145.79.227.7:4293 , piece_id=PAWHOP4S5N2SWCW2NVTGSGGGZDYKVZ52HE5LUXWBC45WTCZ47LXQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.124260658 (609h44m33.995343693s) [noise=true , address=145.79.227.20:4438 , piece_id=HHBPYPKJUJOGJA76IYPO3QBCURQBXSZAMDRI6UGEV4RAGKSAZ3FQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.125012440 (609h44m33.996095485s) [noise=true , address=145.79.227.22:4511 , piece_id=UEOO6Y5AFCQ5RPEFKQHNXIM6DQ6CNVGJ46SU7MLECWVP6NEGV4WQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.132053355 (609h44m34.003136411s) [noise=true , address=145.79.227.39:4014 , piece_id=ZB3AA4NG3QF6PLHH6GTGD7PF7FAXUBOEPWVOIOJ5JRQLAZOVBGQA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.135253870 (609h44m34.006336905s) [noise=true , address=145.79.227.3:4150 , piece_id=LRCVVPU4EXBCU6T4FF4PJUUSJYM4RCR5WPVGAYCAWBFIOE77QQRQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.138185082 (609h44m34.009268117s) [noise=true , address=145.79.227.5:4222 , piece_id=AU6TDEXDLGFCXPSFIKYQ3AM4UXBXVMXZ2RVIILMBX6I6UZWU7CJQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.145248080 (609h44m34.016331116s) [noise=true , address=145.79.227.37:4565 , piece_id=YYTOH6DR725IZRPAGNOOTZCSQEI55FEYKDL4PNF3SYONLMOHQ25A, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.146704714 (609h44m34.017787779s) [noise=true , address=145.79.227.2:4113 , piece_id=A2U3D2C7LEPHVPU3IUKBNNM6BFHBW2AOMRRIZ6XPB2MEIANKPRXQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.312085941 (609h44m34.183168976s) [noise=true , address=77.90.31.16:4735 , piece_id=DDIONJAYZBASQJDP3QPISHXACAFZGOW3EBSMIXNNT2LY4DLYXADQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.436213054 (609h44m34.307296109s) [noise=true , address=45.91.116.14:4189 , piece_id=FAPD7HNXB3TV33NFDF4UFJGHY7AWSPTL3KZHMQ4QQIYZ7ZEMSJGA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.766896134 (609h44m34.637979179s) [noise=true , address=45.91.116.7:4011 , piece_id=4SKDVZH2AHRVLV5362AOOCGLOBIPEK7WGDBMWLGWZB3Q6NNLQPSA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.034120011 (609h44m34.905203046s) [noise=true , address=77.90.31.21:4877 , piece_id=BBOICMOZ3RSNP3I5XX2AQW3B6SURUFUDCTZG25KXP62FSNOSRQUQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.317152980 (609h44m35.188236025s) [noise=true , address=45.91.116.6:4973 , piece_id=IUHID3DDNNPTIAQXX4UCETB5LFFJBYHM7POMA7ALIZYZALICQ73Q, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.317171395 (609h44m35.18825443s) [seg_pos={0 0}, upload=534365]: Segment reached optimal threshold of 13 pieces. Dec 31 01:13:35.317363992 (609h44m35.188447027s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317369673 (609h44m35.188452708s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317497957 (609h44m35.188580992s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317503487 (609h44m35.188586522s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317612284 (609h44m35.188695339s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317643834 (609h44m35.188726869s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317682628 (609h44m35.188765663s) [seg_pos={0 0}, upload=534365]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:13:35.317778912 (609h44m35.188861947s) [upload=534365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:13:35.317784572 (609h44m35.188867607s) [upload=534365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:13:35.615134704 (609h44m35.486217749s) [upload=534365]: Done uploading segments... Dec 31 01:13:35.615159452 (609h44m35.486242497s) [upload=534365]: Upload finished. err: <nil> Duration: 5.819497608s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.06753071s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:15:03 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.06756245s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.534757902s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:14:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.534783531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.........................................X...........] duration: 45.685957022s trace: https://jaeger.dev.storj.io/trace/6692771f1d9bd6b2 FAIL (2025-12-31 01:14:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998406888 (609h37m17.869489933s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998969990 (609h37m17.870053035s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999423213 (609h37m17.870506258s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999952231 (609h37m17.871035276s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000438797 (609h37m17.871521842s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000949088 (609h37m17.872032133s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.003033838 (609h37m17.874116893s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.004847171 (609h37m17.875930206s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.006308694 (609h37m17.877391729s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.008663187 (609h37m17.879746233s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.011451617 (609h37m17.882534662s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.012760589 (609h37m17.883843624s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.014222312 (609h37m17.885305357s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:19.443835684 (609h37m19.314918729s) [upload=534265]: Next returned nil segment Dec 31 01:06:19.443849530 (609h37m19.314932575s) [upload=534265]: Waiting for error group managing segments... Dec 31 01:06:19.844855896 (609h37m19.715938931s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.850883323 (609h37m19.721966368s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.854227971 (609h37m19.725311017s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.989572655 (609h37m19.8606557s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.038160412 (609h37m19.909243457s) [noise=true , address=77.90.31.16:4737 , piece_id=5QISDRHH3CIX3GBOZWFGVI5HCRMR3RMZSCC4KINC44ED2RYP77MA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.045562486 (609h37m19.916645531s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.110196167 (609h37m19.981279202s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.180265638 (609h37m20.051348694s) [noise=true , address=45.91.116.6:4002 , piece_id=5BENLC3SQHUHMN6EZN47GBMUBMERR74MPQF4YCGSMDGYIKGC7PBA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.193916461 (609h37m20.064999506s) [noise=true , address=45.91.116.8:4077 , piece_id=OAMG33FNNBKQAOXCGORIU7LQGKD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.724980742 (609h37m20.596063777s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.767733068 (609h37m20.638816103s) [noise=true , address=145.79.227.30:4782 , piece_id=R2CNB63KEQFO2FVF2DH3LOINRIWKDHQICDSXJZAUG2X4CF2VXIQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.809884670 (609h37m20.680967715s) [noise=true , address=145.79.227.44:4219 , piece_id=ILJTHSDETVN273ML35ABI5ERFLORQGUK7YVCFCMKUSL6OPBC2PDA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.904592756 (609h37m20.775675791s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977540337 (609h37m20.848623382s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977565255 (609h37m20.84864829s) [seg_pos={0 0}, upload=534265]: Segment reached optimal threshold of 14 pieces. Dec 31 01:06:20.977775716 (609h37m20.848858761s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection Dec 31 01:06:20.977784612 (609h37m20.848867647s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.977846340 (609h37m20.848929385s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DPUW1Ej5FLg9yTbhqdWBX8YAryWG82BnVegsCiTRX71pUA24n): protocol: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection Dec 31 01:06:20.977858483 (609h37m20.848941528s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978094993 (609h37m20.849178038s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF): protocol: expected piece hash; context canceled; EOF Dec 31 01:06:20.978105122 (609h37m20.849188167s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978136983 (609h37m20.849220018s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection Dec 31 01:06:20.978148856 (609h37m20.849231901s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978171629 (609h37m20.849254674s) [seg_pos={0 0}, upload=534265]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:06:20.978256441 (609h37m20.849339476s) [upload=534265]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:06:20.978263184 (609h37m20.849346219s) [upload=534265]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:06:21.275269351 (609h37m21.146352406s) [upload=534265]: Done uploading segments... Dec 31 01:06:21.275294678 (609h37m21.146377713s) [upload=534265]: Upload finished. err: <nil> Duration: 45.685996447s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.563277108s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:14:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.563305953s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.040206151s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:14:14 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.040233513s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX.....X.] duration: 5.955976597s trace: https://jaeger.dev.storj.io/trace/5c211acadf0a2f7c FAIL (2025-12-31 01:14:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:13:31.892162068 (609h44m31.763245103s) [upload=534365]: Starting upload Dec 31 01:13:31.892229988 (609h44m31.763313023s) [upload=534365]: Done starting upload Dec 31 01:13:31.892520051 (609h44m31.763603086s) [upload=534365]: Uploading segments... Dec 31 01:13:31.892835572 (609h44m31.763918607s) [upload=534365]: Got next segment. Inline: false Dec 31 01:13:31.892880577 (609h44m31.763963622s) [upload=534365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:13:31.892886969 (609h44m31.763970014s) [upload=534365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:13:32.136582035 (609h44m32.00766508s) [seg_pos={0 0}, upload=534365]: Begin upload segment... Dec 31 01:13:32.136741648 (609h44m32.007824693s) [seg_pos={0 0}, upload=534365]: Done begin upload segment. Dec 31 01:13:32.136787886 (609h44m32.007870921s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.136952009 (609h44m32.008035054s) [noise=true , address=145.79.227.7:4293 , piece_id=PAWHOP4S5N2SWCW2NVTGSGGGZDYKVZ52HE5LUXWBC45WTCZ47LXQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137392508 (609h44m32.008475553s) [noise=true , address=145.79.227.20:4438 , piece_id=HHBPYPKJUJOGJA76IYPO3QBCURQBXSZAMDRI6UGEV4RAGKSAZ3FQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137833196 (609h44m32.008916241s) [noise=true , address=145.79.227.22:4511 , piece_id=UEOO6Y5AFCQ5RPEFKQHNXIM6DQ6CNVGJ46SU7MLECWVP6NEGV4WQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137887390 (609h44m32.008970435s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138202520 (609h44m32.009285555s) [noise=true , address=45.91.116.7:4011 , piece_id=4SKDVZH2AHRVLV5362AOOCGLOBIPEK7WGDBMWLGWZB3Q6NNLQPSA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138443369 (609h44m32.009526414s) [noise=true , address=145.79.227.37:4565 , piece_id=YYTOH6DR725IZRPAGNOOTZCSQEI55FEYKDL4PNF3SYONLMOHQ25A, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138677505 (609h44m32.00976054s) [noise=true , address=145.79.227.39:4014 , piece_id=ZB3AA4NG3QF6PLHH6GTGD7PF7FAXUBOEPWVOIOJ5JRQLAZOVBGQA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138997634 (609h44m32.010080679s) [noise=true , address=145.79.227.5:4222 , piece_id=AU6TDEXDLGFCXPSFIKYQ3AM4UXBXVMXZ2RVIILMBX6I6UZWU7CJQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139179520 (609h44m32.010262565s) [noise=true , address=145.79.227.3:4150 , piece_id=LRCVVPU4EXBCU6T4FF4PJUUSJYM4RCR5WPVGAYCAWBFIOE77QQRQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139293667 (609h44m32.010376712s) [noise=true , address=45.91.116.6:4973 , piece_id=IUHID3DDNNPTIAQXX4UCETB5LFFJBYHM7POMA7ALIZYZALICQ73Q, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139647271 (609h44m32.010730316s) [noise=true , address=77.90.31.16:4735 , piece_id=DDIONJAYZBASQJDP3QPISHXACAFZGOW3EBSMIXNNT2LY4DLYXADQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139782598 (609h44m32.010865643s) [noise=true , address=45.91.116.14:4189 , piece_id=FAPD7HNXB3TV33NFDF4UFJGHY7AWSPTL3KZHMQ4QQIYZ7ZEMSJGA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.140427918 (609h44m32.011510953s) [noise=true , address=145.79.227.2:4113 , piece_id=A2U3D2C7LEPHVPU3IUKBNNM6BFHBW2AOMRRIZ6XPB2MEIANKPRXQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.141112661 (609h44m32.012195706s) [noise=true , address=77.90.31.21:4877 , piece_id=BBOICMOZ3RSNP3I5XX2AQW3B6SURUFUDCTZG25KXP62FSNOSRQUQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.141783057 (609h44m32.012866102s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:33.837053250 (609h44m33.708136285s) [upload=534365]: Next returned nil segment Dec 31 01:13:33.837088717 (609h44m33.708171752s) [upload=534365]: Waiting for error group managing segments... Dec 31 01:13:34.123562999 (609h44m33.994646034s) [noise=true , address=145.79.227.7:4293 , piece_id=PAWHOP4S5N2SWCW2NVTGSGGGZDYKVZ52HE5LUXWBC45WTCZ47LXQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.124260658 (609h44m33.995343693s) [noise=true , address=145.79.227.20:4438 , piece_id=HHBPYPKJUJOGJA76IYPO3QBCURQBXSZAMDRI6UGEV4RAGKSAZ3FQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.125012440 (609h44m33.996095485s) [noise=true , address=145.79.227.22:4511 , piece_id=UEOO6Y5AFCQ5RPEFKQHNXIM6DQ6CNVGJ46SU7MLECWVP6NEGV4WQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.132053355 (609h44m34.003136411s) [noise=true , address=145.79.227.39:4014 , piece_id=ZB3AA4NG3QF6PLHH6GTGD7PF7FAXUBOEPWVOIOJ5JRQLAZOVBGQA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.135253870 (609h44m34.006336905s) [noise=true , address=145.79.227.3:4150 , piece_id=LRCVVPU4EXBCU6T4FF4PJUUSJYM4RCR5WPVGAYCAWBFIOE77QQRQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.138185082 (609h44m34.009268117s) [noise=true , address=145.79.227.5:4222 , piece_id=AU6TDEXDLGFCXPSFIKYQ3AM4UXBXVMXZ2RVIILMBX6I6UZWU7CJQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.145248080 (609h44m34.016331116s) [noise=true , address=145.79.227.37:4565 , piece_id=YYTOH6DR725IZRPAGNOOTZCSQEI55FEYKDL4PNF3SYONLMOHQ25A, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.146704714 (609h44m34.017787779s) [noise=true , address=145.79.227.2:4113 , piece_id=A2U3D2C7LEPHVPU3IUKBNNM6BFHBW2AOMRRIZ6XPB2MEIANKPRXQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.312085941 (609h44m34.183168976s) [noise=true , address=77.90.31.16:4735 , piece_id=DDIONJAYZBASQJDP3QPISHXACAFZGOW3EBSMIXNNT2LY4DLYXADQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.436213054 (609h44m34.307296109s) [noise=true , address=45.91.116.14:4189 , piece_id=FAPD7HNXB3TV33NFDF4UFJGHY7AWSPTL3KZHMQ4QQIYZ7ZEMSJGA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.766896134 (609h44m34.637979179s) [noise=true , address=45.91.116.7:4011 , piece_id=4SKDVZH2AHRVLV5362AOOCGLOBIPEK7WGDBMWLGWZB3Q6NNLQPSA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.034120011 (609h44m34.905203046s) [noise=true , address=77.90.31.21:4877 , piece_id=BBOICMOZ3RSNP3I5XX2AQW3B6SURUFUDCTZG25KXP62FSNOSRQUQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.317152980 (609h44m35.188236025s) [noise=true , address=45.91.116.6:4973 , piece_id=IUHID3DDNNPTIAQXX4UCETB5LFFJBYHM7POMA7ALIZYZALICQ73Q, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.317171395 (609h44m35.18825443s) [seg_pos={0 0}, upload=534365]: Segment reached optimal threshold of 13 pieces. Dec 31 01:13:35.317363992 (609h44m35.188447027s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317369673 (609h44m35.188452708s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317497957 (609h44m35.188580992s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317503487 (609h44m35.188586522s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317612284 (609h44m35.188695339s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317643834 (609h44m35.188726869s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317682628 (609h44m35.188765663s) [seg_pos={0 0}, upload=534365]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:13:35.317778912 (609h44m35.188861947s) [upload=534365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:13:35.317784572 (609h44m35.188867607s) [upload=534365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:13:35.615134704 (609h44m35.486217749s) [upload=534365]: Done uploading segments... Dec 31 01:13:35.615159452 (609h44m35.486242497s) [upload=534365]: Upload finished. err: <nil> Duration: 5.956006524s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.XXX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX.....X] duration: 30.00279558s trace: https://jaeger.dev.storj.io/trace/5c211acadf0a2f7c FAIL (2025-12-31 01:14:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:13:31.892162068 (609h44m31.763245103s) [upload=534365]: Starting upload Dec 31 01:13:31.892229988 (609h44m31.763313023s) [upload=534365]: Done starting upload Dec 31 01:13:31.892520051 (609h44m31.763603086s) [upload=534365]: Uploading segments... Dec 31 01:13:31.892835572 (609h44m31.763918607s) [upload=534365]: Got next segment. Inline: false Dec 31 01:13:31.892880577 (609h44m31.763963622s) [upload=534365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:13:31.892886969 (609h44m31.763970014s) [upload=534365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:13:32.136582035 (609h44m32.00766508s) [seg_pos={0 0}, upload=534365]: Begin upload segment... Dec 31 01:13:32.136741648 (609h44m32.007824693s) [seg_pos={0 0}, upload=534365]: Done begin upload segment. Dec 31 01:13:32.136787886 (609h44m32.007870921s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.136952009 (609h44m32.008035054s) [noise=true , address=145.79.227.7:4293 , piece_id=PAWHOP4S5N2SWCW2NVTGSGGGZDYKVZ52HE5LUXWBC45WTCZ47LXQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137392508 (609h44m32.008475553s) [noise=true , address=145.79.227.20:4438 , piece_id=HHBPYPKJUJOGJA76IYPO3QBCURQBXSZAMDRI6UGEV4RAGKSAZ3FQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137833196 (609h44m32.008916241s) [noise=true , address=145.79.227.22:4511 , piece_id=UEOO6Y5AFCQ5RPEFKQHNXIM6DQ6CNVGJ46SU7MLECWVP6NEGV4WQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137887390 (609h44m32.008970435s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138202520 (609h44m32.009285555s) [noise=true , address=45.91.116.7:4011 , piece_id=4SKDVZH2AHRVLV5362AOOCGLOBIPEK7WGDBMWLGWZB3Q6NNLQPSA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138443369 (609h44m32.009526414s) [noise=true , address=145.79.227.37:4565 , piece_id=YYTOH6DR725IZRPAGNOOTZCSQEI55FEYKDL4PNF3SYONLMOHQ25A, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138677505 (609h44m32.00976054s) [noise=true , address=145.79.227.39:4014 , piece_id=ZB3AA4NG3QF6PLHH6GTGD7PF7FAXUBOEPWVOIOJ5JRQLAZOVBGQA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138997634 (609h44m32.010080679s) [noise=true , address=145.79.227.5:4222 , piece_id=AU6TDEXDLGFCXPSFIKYQ3AM4UXBXVMXZ2RVIILMBX6I6UZWU7CJQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139179520 (609h44m32.010262565s) [noise=true , address=145.79.227.3:4150 , piece_id=LRCVVPU4EXBCU6T4FF4PJUUSJYM4RCR5WPVGAYCAWBFIOE77QQRQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139293667 (609h44m32.010376712s) [noise=true , address=45.91.116.6:4973 , piece_id=IUHID3DDNNPTIAQXX4UCETB5LFFJBYHM7POMA7ALIZYZALICQ73Q, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139647271 (609h44m32.010730316s) [noise=true , address=77.90.31.16:4735 , piece_id=DDIONJAYZBASQJDP3QPISHXACAFZGOW3EBSMIXNNT2LY4DLYXADQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139782598 (609h44m32.010865643s) [noise=true , address=45.91.116.14:4189 , piece_id=FAPD7HNXB3TV33NFDF4UFJGHY7AWSPTL3KZHMQ4QQIYZ7ZEMSJGA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.140427918 (609h44m32.011510953s) [noise=true , address=145.79.227.2:4113 , piece_id=A2U3D2C7LEPHVPU3IUKBNNM6BFHBW2AOMRRIZ6XPB2MEIANKPRXQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.141112661 (609h44m32.012195706s) [noise=true , address=77.90.31.21:4877 , piece_id=BBOICMOZ3RSNP3I5XX2AQW3B6SURUFUDCTZG25KXP62FSNOSRQUQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.141783057 (609h44m32.012866102s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:33.837053250 (609h44m33.708136285s) [upload=534365]: Next returned nil segment Dec 31 01:13:33.837088717 (609h44m33.708171752s) [upload=534365]: Waiting for error group managing segments... Dec 31 01:13:34.123562999 (609h44m33.994646034s) [noise=true , address=145.79.227.7:4293 , piece_id=PAWHOP4S5N2SWCW2NVTGSGGGZDYKVZ52HE5LUXWBC45WTCZ47LXQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.124260658 (609h44m33.995343693s) [noise=true , address=145.79.227.20:4438 , piece_id=HHBPYPKJUJOGJA76IYPO3QBCURQBXSZAMDRI6UGEV4RAGKSAZ3FQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.125012440 (609h44m33.996095485s) [noise=true , address=145.79.227.22:4511 , piece_id=UEOO6Y5AFCQ5RPEFKQHNXIM6DQ6CNVGJ46SU7MLECWVP6NEGV4WQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.132053355 (609h44m34.003136411s) [noise=true , address=145.79.227.39:4014 , piece_id=ZB3AA4NG3QF6PLHH6GTGD7PF7FAXUBOEPWVOIOJ5JRQLAZOVBGQA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.135253870 (609h44m34.006336905s) [noise=true , address=145.79.227.3:4150 , piece_id=LRCVVPU4EXBCU6T4FF4PJUUSJYM4RCR5WPVGAYCAWBFIOE77QQRQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.138185082 (609h44m34.009268117s) [noise=true , address=145.79.227.5:4222 , piece_id=AU6TDEXDLGFCXPSFIKYQ3AM4UXBXVMXZ2RVIILMBX6I6UZWU7CJQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.145248080 (609h44m34.016331116s) [noise=true , address=145.79.227.37:4565 , piece_id=YYTOH6DR725IZRPAGNOOTZCSQEI55FEYKDL4PNF3SYONLMOHQ25A, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.146704714 (609h44m34.017787779s) [noise=true , address=145.79.227.2:4113 , piece_id=A2U3D2C7LEPHVPU3IUKBNNM6BFHBW2AOMRRIZ6XPB2MEIANKPRXQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.312085941 (609h44m34.183168976s) [noise=true , address=77.90.31.16:4735 , piece_id=DDIONJAYZBASQJDP3QPISHXACAFZGOW3EBSMIXNNT2LY4DLYXADQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.436213054 (609h44m34.307296109s) [noise=true , address=45.91.116.14:4189 , piece_id=FAPD7HNXB3TV33NFDF4UFJGHY7AWSPTL3KZHMQ4QQIYZ7ZEMSJGA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.766896134 (609h44m34.637979179s) [noise=true , address=45.91.116.7:4011 , piece_id=4SKDVZH2AHRVLV5362AOOCGLOBIPEK7WGDBMWLGWZB3Q6NNLQPSA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.034120011 (609h44m34.905203046s) [noise=true , address=77.90.31.21:4877 , piece_id=BBOICMOZ3RSNP3I5XX2AQW3B6SURUFUDCTZG25KXP62FSNOSRQUQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.317152980 (609h44m35.188236025s) [noise=true , address=45.91.116.6:4973 , piece_id=IUHID3DDNNPTIAQXX4UCETB5LFFJBYHM7POMA7ALIZYZALICQ73Q, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.317171395 (609h44m35.18825443s) [seg_pos={0 0}, upload=534365]: Segment reached optimal threshold of 13 pieces. Dec 31 01:13:35.317363992 (609h44m35.188447027s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317369673 (609h44m35.188452708s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317497957 (609h44m35.188580992s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317503487 (609h44m35.188586522s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317612284 (609h44m35.188695339s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317643834 (609h44m35.188726869s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317682628 (609h44m35.188765663s) [seg_pos={0 0}, upload=534365]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:13:35.317778912 (609h44m35.188861947s) [upload=534365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:13:35.317784572 (609h44m35.188867607s) [upload=534365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:13:35.615134704 (609h44m35.486217749s) [upload=534365]: Done uploading segments... Dec 31 01:13:35.615159452 (609h44m35.486242497s) [upload=534365]: Upload finished. err: <nil> Duration: 30.002822792s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.261543666s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:13:42 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.261575477s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.........................................X..........] duration: 28.177976072s trace: https://jaeger.dev.storj.io/trace/6692771f1d9bd6b2 FAIL (2025-12-31 01:13:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998406888 (609h37m17.869489933s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998969990 (609h37m17.870053035s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999423213 (609h37m17.870506258s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999952231 (609h37m17.871035276s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000438797 (609h37m17.871521842s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000949088 (609h37m17.872032133s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.003033838 (609h37m17.874116893s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.004847171 (609h37m17.875930206s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.006308694 (609h37m17.877391729s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.008663187 (609h37m17.879746233s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.011451617 (609h37m17.882534662s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.012760589 (609h37m17.883843624s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.014222312 (609h37m17.885305357s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:19.443835684 (609h37m19.314918729s) [upload=534265]: Next returned nil segment Dec 31 01:06:19.443849530 (609h37m19.314932575s) [upload=534265]: Waiting for error group managing segments... Dec 31 01:06:19.844855896 (609h37m19.715938931s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.850883323 (609h37m19.721966368s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.854227971 (609h37m19.725311017s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.989572655 (609h37m19.8606557s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.038160412 (609h37m19.909243457s) [noise=true , address=77.90.31.16:4737 , piece_id=5QISDRHH3CIX3GBOZWFGVI5HCRMR3RMZSCC4KINC44ED2RYP77MA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.045562486 (609h37m19.916645531s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.110196167 (609h37m19.981279202s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.180265638 (609h37m20.051348694s) [noise=true , address=45.91.116.6:4002 , piece_id=5BENLC3SQHUHMN6EZN47GBMUBMERR74MPQF4YCGSMDGYIKGC7PBA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.193916461 (609h37m20.064999506s) [noise=true , address=45.91.116.8:4077 , piece_id=OAMG33FNNBKQAOXCGORIU7LQGKD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.724980742 (609h37m20.596063777s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.767733068 (609h37m20.638816103s) [noise=true , address=145.79.227.30:4782 , piece_id=R2CNB63KEQFO2FVF2DH3LOINRIWKDHQICDSXJZAUG2X4CF2VXIQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.809884670 (609h37m20.680967715s) [noise=true , address=145.79.227.44:4219 , piece_id=ILJTHSDETVN273ML35ABI5ERFLORQGUK7YVCFCMKUSL6OPBC2PDA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.904592756 (609h37m20.775675791s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977540337 (609h37m20.848623382s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977565255 (609h37m20.84864829s) [seg_pos={0 0}, upload=534265]: Segment reached optimal threshold of 14 pieces. Dec 31 01:06:20.977775716 (609h37m20.848858761s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection Dec 31 01:06:20.977784612 (609h37m20.848867647s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.977846340 (609h37m20.848929385s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DPUW1Ej5FLg9yTbhqdWBX8YAryWG82BnVegsCiTRX71pUA24n): protocol: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection Dec 31 01:06:20.977858483 (609h37m20.848941528s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978094993 (609h37m20.849178038s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF): protocol: expected piece hash; context canceled; EOF Dec 31 01:06:20.978105122 (609h37m20.849188167s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978136983 (609h37m20.849220018s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection Dec 31 01:06:20.978148856 (609h37m20.849231901s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978171629 (609h37m20.849254674s) [seg_pos={0 0}, upload=534265]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:06:20.978256441 (609h37m20.849339476s) [upload=534265]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:06:20.978263184 (609h37m20.849346219s) [upload=534265]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:06:21.275269351 (609h37m21.146352406s) [upload=534265]: Done uploading segments... Dec 31 01:06:21.275294678 (609h37m21.146377713s) [upload=534265]: Upload finished. err: <nil> Duration: 28.178002572s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.403227113s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:13:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.403260656s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.XXX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX.....] duration: 6.10499026s trace: https://jaeger.dev.storj.io/trace/302875e9b5456820 FAIL (2025-12-31 01:13:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:09:30.952291082 (609h40m30.823374117s) [upload=534310]: Starting upload Dec 31 01:09:30.952342189 (609h40m30.823425234s) [upload=534310]: Done starting upload Dec 31 01:09:30.952511982 (609h40m30.823595017s) [upload=534310]: Uploading segments... Dec 31 01:09:30.952552770 (609h40m30.823635805s) [upload=534310]: Got next segment. Inline: false Dec 31 01:09:30.952579671 (609h40m30.823662706s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:09:30.952584220 (609h40m30.823667255s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:09:31.200060645 (609h40m31.07114369s) [seg_pos={0 0}, upload=534310]: Begin upload segment... Dec 31 01:09:31.200115720 (609h40m31.071198756s) [seg_pos={0 0}, upload=534310]: Done begin upload segment. Dec 31 01:09:31.200160485 (609h40m31.07124353s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.200183600 (609h40m31.071266635s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201328900 (609h40m31.072411945s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201655632 (609h40m31.072738678s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202001441 (609h40m31.073084476s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202298125 (609h40m31.07338116s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202753022 (609h40m31.073836067s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203062341 (609h40m31.074145386s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203432997 (609h40m31.074516032s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203781821 (609h40m31.074864866s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204750115 (609h40m31.07583315s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204981074 (609h40m31.076064119s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206060129 (609h40m31.077143164s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206134350 (609h40m31.077217385s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.207278809 (609h40m31.078361845s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.208923160 (609h40m31.080006205s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:32.904529566 (609h40m32.775612611s) [upload=534310]: Next returned nil segment Dec 31 01:09:32.904573228 (609h40m32.775656273s) [upload=534310]: Waiting for error group managing segments... Dec 31 01:09:33.177751323 (609h40m33.048834358s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.188115618 (609h40m33.059198663s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.193468411 (609h40m33.064551456s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.199886171 (609h40m33.070969206s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.203595614 (609h40m33.074678649s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.205080291 (609h40m33.076163326s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.209178975 (609h40m33.08026201s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.217865655 (609h40m33.0889487s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.219876184 (609h40m33.090959219s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.358860787 (609h40m33.229943832s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.383597680 (609h40m33.254680715s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.463911717 (609h40m33.334994762s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535677719 (609h40m33.406760754s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535698058 (609h40m33.406781103s) [seg_pos={0 0}, upload=534310]: Segment reached optimal threshold of 13 pieces. Dec 31 01:09:33.535862290 (609h40m33.406945335s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.535869223 (609h40m33.406952258s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.535958804 (609h40m33.407041849s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection Dec 31 01:09:33.535974513 (609h40m33.407057548s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536069374 (609h40m33.407152459s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.536085155 (609h40m33.40716819s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536110973 (609h40m33.407194018s) [seg_pos={0 0}, upload=534310]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:09:33.536258765 (609h40m33.407341811s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:09:33.536265258 (609h40m33.407348303s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:09:33.838979262 (609h40m33.710062308s) [upload=534310]: Done uploading segments... Dec 31 01:09:33.839001435 (609h40m33.71008448s) [upload=534310]: Upload finished. err: <nil> Duration: 6.1050164s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.002781883s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:13:03 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.002817261s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.106871469s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:12:41 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.106892981s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.........................................X.........] duration: 34.966114708s trace: https://jaeger.dev.storj.io/trace/6692771f1d9bd6b2 FAIL (2025-12-31 01:12:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998406888 (609h37m17.869489933s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998969990 (609h37m17.870053035s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999423213 (609h37m17.870506258s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999952231 (609h37m17.871035276s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000438797 (609h37m17.871521842s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000949088 (609h37m17.872032133s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.003033838 (609h37m17.874116893s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.004847171 (609h37m17.875930206s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.006308694 (609h37m17.877391729s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.008663187 (609h37m17.879746233s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.011451617 (609h37m17.882534662s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.012760589 (609h37m17.883843624s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.014222312 (609h37m17.885305357s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:19.443835684 (609h37m19.314918729s) [upload=534265]: Next returned nil segment Dec 31 01:06:19.443849530 (609h37m19.314932575s) [upload=534265]: Waiting for error group managing segments... Dec 31 01:06:19.844855896 (609h37m19.715938931s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.850883323 (609h37m19.721966368s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.854227971 (609h37m19.725311017s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.989572655 (609h37m19.8606557s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.038160412 (609h37m19.909243457s) [noise=true , address=77.90.31.16:4737 , piece_id=5QISDRHH3CIX3GBOZWFGVI5HCRMR3RMZSCC4KINC44ED2RYP77MA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.045562486 (609h37m19.916645531s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.110196167 (609h37m19.981279202s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.180265638 (609h37m20.051348694s) [noise=true , address=45.91.116.6:4002 , piece_id=5BENLC3SQHUHMN6EZN47GBMUBMERR74MPQF4YCGSMDGYIKGC7PBA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.193916461 (609h37m20.064999506s) [noise=true , address=45.91.116.8:4077 , piece_id=OAMG33FNNBKQAOXCGORIU7LQGKD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.724980742 (609h37m20.596063777s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.767733068 (609h37m20.638816103s) [noise=true , address=145.79.227.30:4782 , piece_id=R2CNB63KEQFO2FVF2DH3LOINRIWKDHQICDSXJZAUG2X4CF2VXIQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.809884670 (609h37m20.680967715s) [noise=true , address=145.79.227.44:4219 , piece_id=ILJTHSDETVN273ML35ABI5ERFLORQGUK7YVCFCMKUSL6OPBC2PDA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.904592756 (609h37m20.775675791s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977540337 (609h37m20.848623382s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977565255 (609h37m20.84864829s) [seg_pos={0 0}, upload=534265]: Segment reached optimal threshold of 14 pieces. Dec 31 01:06:20.977775716 (609h37m20.848858761s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection Dec 31 01:06:20.977784612 (609h37m20.848867647s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.977846340 (609h37m20.848929385s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DPUW1Ej5FLg9yTbhqdWBX8YAryWG82BnVegsCiTRX71pUA24n): protocol: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection Dec 31 01:06:20.977858483 (609h37m20.848941528s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978094993 (609h37m20.849178038s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF): protocol: expected piece hash; context canceled; EOF Dec 31 01:06:20.978105122 (609h37m20.849188167s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978136983 (609h37m20.849220018s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection Dec 31 01:06:20.978148856 (609h37m20.849231901s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978171629 (609h37m20.849254674s) [seg_pos={0 0}, upload=534265]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:06:20.978256441 (609h37m20.849339476s) [upload=534265]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:06:20.978263184 (609h37m20.849346219s) [upload=534265]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:06:21.275269351 (609h37m21.146352406s) [upload=534265]: Done uploading segments... Dec 31 01:06:21.275294678 (609h37m21.146377713s) [upload=534265]: Upload finished. err: <nil> Duration: 34.966160525s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.613274869s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:12:23 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.613302201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX.XXX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX....] duration: 6.113872802s trace: https://jaeger.dev.storj.io/trace/302875e9b5456820 FAIL (2025-12-31 01:12:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:09:30.952291082 (609h40m30.823374117s) [upload=534310]: Starting upload Dec 31 01:09:30.952342189 (609h40m30.823425234s) [upload=534310]: Done starting upload Dec 31 01:09:30.952511982 (609h40m30.823595017s) [upload=534310]: Uploading segments... Dec 31 01:09:30.952552770 (609h40m30.823635805s) [upload=534310]: Got next segment. Inline: false Dec 31 01:09:30.952579671 (609h40m30.823662706s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:09:30.952584220 (609h40m30.823667255s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:09:31.200060645 (609h40m31.07114369s) [seg_pos={0 0}, upload=534310]: Begin upload segment... Dec 31 01:09:31.200115720 (609h40m31.071198756s) [seg_pos={0 0}, upload=534310]: Done begin upload segment. Dec 31 01:09:31.200160485 (609h40m31.07124353s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.200183600 (609h40m31.071266635s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201328900 (609h40m31.072411945s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201655632 (609h40m31.072738678s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202001441 (609h40m31.073084476s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202298125 (609h40m31.07338116s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202753022 (609h40m31.073836067s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203062341 (609h40m31.074145386s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203432997 (609h40m31.074516032s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203781821 (609h40m31.074864866s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204750115 (609h40m31.07583315s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204981074 (609h40m31.076064119s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206060129 (609h40m31.077143164s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206134350 (609h40m31.077217385s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.207278809 (609h40m31.078361845s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.208923160 (609h40m31.080006205s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:32.904529566 (609h40m32.775612611s) [upload=534310]: Next returned nil segment Dec 31 01:09:32.904573228 (609h40m32.775656273s) [upload=534310]: Waiting for error group managing segments... Dec 31 01:09:33.177751323 (609h40m33.048834358s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.188115618 (609h40m33.059198663s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.193468411 (609h40m33.064551456s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.199886171 (609h40m33.070969206s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.203595614 (609h40m33.074678649s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.205080291 (609h40m33.076163326s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.209178975 (609h40m33.08026201s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.217865655 (609h40m33.0889487s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.219876184 (609h40m33.090959219s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.358860787 (609h40m33.229943832s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.383597680 (609h40m33.254680715s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.463911717 (609h40m33.334994762s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535677719 (609h40m33.406760754s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535698058 (609h40m33.406781103s) [seg_pos={0 0}, upload=534310]: Segment reached optimal threshold of 13 pieces. Dec 31 01:09:33.535862290 (609h40m33.406945335s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.535869223 (609h40m33.406952258s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.535958804 (609h40m33.407041849s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection Dec 31 01:09:33.535974513 (609h40m33.407057548s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536069374 (609h40m33.407152459s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.536085155 (609h40m33.40716819s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536110973 (609h40m33.407194018s) [seg_pos={0 0}, upload=534310]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:09:33.536258765 (609h40m33.407341811s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:09:33.536265258 (609h40m33.407348303s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:09:33.838979262 (609h40m33.710062308s) [upload=534310]: Done uploading segments... Dec 31 01:09:33.839001435 (609h40m33.71008448s) [upload=534310]: Upload finished. err: <nil> Duration: 6.113898622s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.455957989s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:12:05 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.455984287s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.627391583s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:11:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.627418112s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.........................................X........] duration: 15.507208865s trace: https://jaeger.dev.storj.io/trace/6692771f1d9bd6b2 FAIL (2025-12-31 01:11:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998406888 (609h37m17.869489933s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998969990 (609h37m17.870053035s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999423213 (609h37m17.870506258s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999952231 (609h37m17.871035276s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000438797 (609h37m17.871521842s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000949088 (609h37m17.872032133s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.003033838 (609h37m17.874116893s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.004847171 (609h37m17.875930206s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.006308694 (609h37m17.877391729s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.008663187 (609h37m17.879746233s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.011451617 (609h37m17.882534662s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.012760589 (609h37m17.883843624s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.014222312 (609h37m17.885305357s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:19.443835684 (609h37m19.314918729s) [upload=534265]: Next returned nil segment Dec 31 01:06:19.443849530 (609h37m19.314932575s) [upload=534265]: Waiting for error group managing segments... Dec 31 01:06:19.844855896 (609h37m19.715938931s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.850883323 (609h37m19.721966368s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.854227971 (609h37m19.725311017s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.989572655 (609h37m19.8606557s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.038160412 (609h37m19.909243457s) [noise=true , address=77.90.31.16:4737 , piece_id=5QISDRHH3CIX3GBOZWFGVI5HCRMR3RMZSCC4KINC44ED2RYP77MA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.045562486 (609h37m19.916645531s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.110196167 (609h37m19.981279202s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.180265638 (609h37m20.051348694s) [noise=true , address=45.91.116.6:4002 , piece_id=5BENLC3SQHUHMN6EZN47GBMUBMERR74MPQF4YCGSMDGYIKGC7PBA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.193916461 (609h37m20.064999506s) [noise=true , address=45.91.116.8:4077 , piece_id=OAMG33FNNBKQAOXCGORIU7LQGKD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.724980742 (609h37m20.596063777s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.767733068 (609h37m20.638816103s) [noise=true , address=145.79.227.30:4782 , piece_id=R2CNB63KEQFO2FVF2DH3LOINRIWKDHQICDSXJZAUG2X4CF2VXIQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.809884670 (609h37m20.680967715s) [noise=true , address=145.79.227.44:4219 , piece_id=ILJTHSDETVN273ML35ABI5ERFLORQGUK7YVCFCMKUSL6OPBC2PDA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.904592756 (609h37m20.775675791s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977540337 (609h37m20.848623382s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977565255 (609h37m20.84864829s) [seg_pos={0 0}, upload=534265]: Segment reached optimal threshold of 14 pieces. Dec 31 01:06:20.977775716 (609h37m20.848858761s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection Dec 31 01:06:20.977784612 (609h37m20.848867647s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.977846340 (609h37m20.848929385s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DPUW1Ej5FLg9yTbhqdWBX8YAryWG82BnVegsCiTRX71pUA24n): protocol: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection Dec 31 01:06:20.977858483 (609h37m20.848941528s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978094993 (609h37m20.849178038s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF): protocol: expected piece hash; context canceled; EOF Dec 31 01:06:20.978105122 (609h37m20.849188167s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978136983 (609h37m20.849220018s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection Dec 31 01:06:20.978148856 (609h37m20.849231901s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978171629 (609h37m20.849254674s) [seg_pos={0 0}, upload=534265]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:06:20.978256441 (609h37m20.849339476s) [upload=534265]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:06:20.978263184 (609h37m20.849346219s) [upload=534265]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:06:21.275269351 (609h37m21.146352406s) [upload=534265]: Done uploading segments... Dec 31 01:06:21.275294678 (609h37m21.146377713s) [upload=534265]: Upload finished. err: <nil> Duration: 15.507233792s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX.XXX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX...] duration: 5.91640484s trace: https://jaeger.dev.storj.io/trace/302875e9b5456820 FAIL (2025-12-31 01:11:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:09:30.952291082 (609h40m30.823374117s) [upload=534310]: Starting upload Dec 31 01:09:30.952342189 (609h40m30.823425234s) [upload=534310]: Done starting upload Dec 31 01:09:30.952511982 (609h40m30.823595017s) [upload=534310]: Uploading segments... Dec 31 01:09:30.952552770 (609h40m30.823635805s) [upload=534310]: Got next segment. Inline: false Dec 31 01:09:30.952579671 (609h40m30.823662706s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:09:30.952584220 (609h40m30.823667255s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:09:31.200060645 (609h40m31.07114369s) [seg_pos={0 0}, upload=534310]: Begin upload segment... Dec 31 01:09:31.200115720 (609h40m31.071198756s) [seg_pos={0 0}, upload=534310]: Done begin upload segment. Dec 31 01:09:31.200160485 (609h40m31.07124353s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.200183600 (609h40m31.071266635s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201328900 (609h40m31.072411945s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201655632 (609h40m31.072738678s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202001441 (609h40m31.073084476s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202298125 (609h40m31.07338116s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202753022 (609h40m31.073836067s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203062341 (609h40m31.074145386s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203432997 (609h40m31.074516032s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203781821 (609h40m31.074864866s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204750115 (609h40m31.07583315s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204981074 (609h40m31.076064119s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206060129 (609h40m31.077143164s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206134350 (609h40m31.077217385s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.207278809 (609h40m31.078361845s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.208923160 (609h40m31.080006205s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:32.904529566 (609h40m32.775612611s) [upload=534310]: Next returned nil segment Dec 31 01:09:32.904573228 (609h40m32.775656273s) [upload=534310]: Waiting for error group managing segments... Dec 31 01:09:33.177751323 (609h40m33.048834358s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.188115618 (609h40m33.059198663s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.193468411 (609h40m33.064551456s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.199886171 (609h40m33.070969206s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.203595614 (609h40m33.074678649s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.205080291 (609h40m33.076163326s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.209178975 (609h40m33.08026201s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.217865655 (609h40m33.0889487s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.219876184 (609h40m33.090959219s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.358860787 (609h40m33.229943832s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.383597680 (609h40m33.254680715s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.463911717 (609h40m33.334994762s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535677719 (609h40m33.406760754s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535698058 (609h40m33.406781103s) [seg_pos={0 0}, upload=534310]: Segment reached optimal threshold of 13 pieces. Dec 31 01:09:33.535862290 (609h40m33.406945335s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.535869223 (609h40m33.406952258s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.535958804 (609h40m33.407041849s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection Dec 31 01:09:33.535974513 (609h40m33.407057548s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536069374 (609h40m33.407152459s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.536085155 (609h40m33.40716819s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536110973 (609h40m33.407194018s) [seg_pos={0 0}, upload=534310]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:09:33.536258765 (609h40m33.407341811s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:09:33.536265258 (609h40m33.407348303s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:09:33.838979262 (609h40m33.710062308s) [upload=534310]: Done uploading segments... Dec 31 01:09:33.839001435 (609h40m33.71008448s) [upload=534310]: Upload finished. err: <nil> Duration: 5.916434547s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.493302312s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:11:30 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.493329182s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.581330897s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:11:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.581356716s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.186992327s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:11:02 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.187016692s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.561963271s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:10:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.562027494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.........................................X.......] duration: 15.342234982s trace: https://jaeger.dev.storj.io/trace/6692771f1d9bd6b2 FAIL (2025-12-31 01:10:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998406888 (609h37m17.869489933s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998969990 (609h37m17.870053035s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999423213 (609h37m17.870506258s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999952231 (609h37m17.871035276s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000438797 (609h37m17.871521842s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000949088 (609h37m17.872032133s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.003033838 (609h37m17.874116893s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.004847171 (609h37m17.875930206s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.006308694 (609h37m17.877391729s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.008663187 (609h37m17.879746233s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.011451617 (609h37m17.882534662s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.012760589 (609h37m17.883843624s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.014222312 (609h37m17.885305357s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:19.443835684 (609h37m19.314918729s) [upload=534265]: Next returned nil segment Dec 31 01:06:19.443849530 (609h37m19.314932575s) [upload=534265]: Waiting for error group managing segments... Dec 31 01:06:19.844855896 (609h37m19.715938931s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.850883323 (609h37m19.721966368s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.854227971 (609h37m19.725311017s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.989572655 (609h37m19.8606557s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.038160412 (609h37m19.909243457s) [noise=true , address=77.90.31.16:4737 , piece_id=5QISDRHH3CIX3GBOZWFGVI5HCRMR3RMZSCC4KINC44ED2RYP77MA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.045562486 (609h37m19.916645531s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.110196167 (609h37m19.981279202s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.180265638 (609h37m20.051348694s) [noise=true , address=45.91.116.6:4002 , piece_id=5BENLC3SQHUHMN6EZN47GBMUBMERR74MPQF4YCGSMDGYIKGC7PBA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.193916461 (609h37m20.064999506s) [noise=true , address=45.91.116.8:4077 , piece_id=OAMG33FNNBKQAOXCGORIU7LQGKD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.724980742 (609h37m20.596063777s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.767733068 (609h37m20.638816103s) [noise=true , address=145.79.227.30:4782 , piece_id=R2CNB63KEQFO2FVF2DH3LOINRIWKDHQICDSXJZAUG2X4CF2VXIQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.809884670 (609h37m20.680967715s) [noise=true , address=145.79.227.44:4219 , piece_id=ILJTHSDETVN273ML35ABI5ERFLORQGUK7YVCFCMKUSL6OPBC2PDA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.904592756 (609h37m20.775675791s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977540337 (609h37m20.848623382s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977565255 (609h37m20.84864829s) [seg_pos={0 0}, upload=534265]: Segment reached optimal threshold of 14 pieces. Dec 31 01:06:20.977775716 (609h37m20.848858761s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection Dec 31 01:06:20.977784612 (609h37m20.848867647s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.977846340 (609h37m20.848929385s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DPUW1Ej5FLg9yTbhqdWBX8YAryWG82BnVegsCiTRX71pUA24n): protocol: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection Dec 31 01:06:20.977858483 (609h37m20.848941528s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978094993 (609h37m20.849178038s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF): protocol: expected piece hash; context canceled; EOF Dec 31 01:06:20.978105122 (609h37m20.849188167s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978136983 (609h37m20.849220018s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection Dec 31 01:06:20.978148856 (609h37m20.849231901s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978171629 (609h37m20.849254674s) [seg_pos={0 0}, upload=534265]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:06:20.978256441 (609h37m20.849339476s) [upload=534265]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:06:20.978263184 (609h37m20.849346219s) [upload=534265]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:06:21.275269351 (609h37m21.146352406s) [upload=534265]: Done uploading segments... Dec 31 01:06:21.275294678 (609h37m21.146377713s) [upload=534265]: Upload finished. err: <nil> Duration: 15.342258707s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.147659494s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:10:31 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.147687146s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..XX.XXX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX..] duration: 6.44768836s trace: https://jaeger.dev.storj.io/trace/302875e9b5456820 FAIL (2025-12-31 01:10:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:09:30.952291082 (609h40m30.823374117s) [upload=534310]: Starting upload Dec 31 01:09:30.952342189 (609h40m30.823425234s) [upload=534310]: Done starting upload Dec 31 01:09:30.952511982 (609h40m30.823595017s) [upload=534310]: Uploading segments... Dec 31 01:09:30.952552770 (609h40m30.823635805s) [upload=534310]: Got next segment. Inline: false Dec 31 01:09:30.952579671 (609h40m30.823662706s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:09:30.952584220 (609h40m30.823667255s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:09:31.200060645 (609h40m31.07114369s) [seg_pos={0 0}, upload=534310]: Begin upload segment... Dec 31 01:09:31.200115720 (609h40m31.071198756s) [seg_pos={0 0}, upload=534310]: Done begin upload segment. Dec 31 01:09:31.200160485 (609h40m31.07124353s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.200183600 (609h40m31.071266635s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201328900 (609h40m31.072411945s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201655632 (609h40m31.072738678s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202001441 (609h40m31.073084476s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202298125 (609h40m31.07338116s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202753022 (609h40m31.073836067s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203062341 (609h40m31.074145386s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203432997 (609h40m31.074516032s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203781821 (609h40m31.074864866s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204750115 (609h40m31.07583315s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204981074 (609h40m31.076064119s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206060129 (609h40m31.077143164s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206134350 (609h40m31.077217385s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.207278809 (609h40m31.078361845s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.208923160 (609h40m31.080006205s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:32.904529566 (609h40m32.775612611s) [upload=534310]: Next returned nil segment Dec 31 01:09:32.904573228 (609h40m32.775656273s) [upload=534310]: Waiting for error group managing segments... Dec 31 01:09:33.177751323 (609h40m33.048834358s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.188115618 (609h40m33.059198663s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.193468411 (609h40m33.064551456s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.199886171 (609h40m33.070969206s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.203595614 (609h40m33.074678649s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.205080291 (609h40m33.076163326s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.209178975 (609h40m33.08026201s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.217865655 (609h40m33.0889487s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.219876184 (609h40m33.090959219s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.358860787 (609h40m33.229943832s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.383597680 (609h40m33.254680715s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.463911717 (609h40m33.334994762s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535677719 (609h40m33.406760754s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535698058 (609h40m33.406781103s) [seg_pos={0 0}, upload=534310]: Segment reached optimal threshold of 13 pieces. Dec 31 01:09:33.535862290 (609h40m33.406945335s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.535869223 (609h40m33.406952258s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.535958804 (609h40m33.407041849s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection Dec 31 01:09:33.535974513 (609h40m33.407057548s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536069374 (609h40m33.407152459s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.536085155 (609h40m33.40716819s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536110973 (609h40m33.407194018s) [seg_pos={0 0}, upload=534310]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:09:33.536258765 (609h40m33.407341811s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:09:33.536265258 (609h40m33.407348303s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:09:33.838979262 (609h40m33.710062308s) [upload=534310]: Done uploading segments... Dec 31 01:09:33.839001435 (609h40m33.71008448s) [upload=534310]: Upload finished. err: <nil> Duration: 6.447715542s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.459328572s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:10:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.459355945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.........................................X......] duration: 32.430480581s trace: https://jaeger.dev.storj.io/trace/6692771f1d9bd6b2 FAIL (2025-12-31 01:10:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998406888 (609h37m17.869489933s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998969990 (609h37m17.870053035s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999423213 (609h37m17.870506258s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999952231 (609h37m17.871035276s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000438797 (609h37m17.871521842s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000949088 (609h37m17.872032133s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.003033838 (609h37m17.874116893s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.004847171 (609h37m17.875930206s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.006308694 (609h37m17.877391729s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.008663187 (609h37m17.879746233s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.011451617 (609h37m17.882534662s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.012760589 (609h37m17.883843624s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.014222312 (609h37m17.885305357s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:19.443835684 (609h37m19.314918729s) [upload=534265]: Next returned nil segment Dec 31 01:06:19.443849530 (609h37m19.314932575s) [upload=534265]: Waiting for error group managing segments... Dec 31 01:06:19.844855896 (609h37m19.715938931s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.850883323 (609h37m19.721966368s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.854227971 (609h37m19.725311017s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.989572655 (609h37m19.8606557s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.038160412 (609h37m19.909243457s) [noise=true , address=77.90.31.16:4737 , piece_id=5QISDRHH3CIX3GBOZWFGVI5HCRMR3RMZSCC4KINC44ED2RYP77MA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.045562486 (609h37m19.916645531s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.110196167 (609h37m19.981279202s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.180265638 (609h37m20.051348694s) [noise=true , address=45.91.116.6:4002 , piece_id=5BENLC3SQHUHMN6EZN47GBMUBMERR74MPQF4YCGSMDGYIKGC7PBA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.193916461 (609h37m20.064999506s) [noise=true , address=45.91.116.8:4077 , piece_id=OAMG33FNNBKQAOXCGORIU7LQGKD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.724980742 (609h37m20.596063777s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.767733068 (609h37m20.638816103s) [noise=true , address=145.79.227.30:4782 , piece_id=R2CNB63KEQFO2FVF2DH3LOINRIWKDHQICDSXJZAUG2X4CF2VXIQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.809884670 (609h37m20.680967715s) [noise=true , address=145.79.227.44:4219 , piece_id=ILJTHSDETVN273ML35ABI5ERFLORQGUK7YVCFCMKUSL6OPBC2PDA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.904592756 (609h37m20.775675791s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977540337 (609h37m20.848623382s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977565255 (609h37m20.84864829s) [seg_pos={0 0}, upload=534265]: Segment reached optimal threshold of 14 pieces. Dec 31 01:06:20.977775716 (609h37m20.848858761s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection Dec 31 01:06:20.977784612 (609h37m20.848867647s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.977846340 (609h37m20.848929385s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DPUW1Ej5FLg9yTbhqdWBX8YAryWG82BnVegsCiTRX71pUA24n): protocol: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection Dec 31 01:06:20.977858483 (609h37m20.848941528s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978094993 (609h37m20.849178038s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF): protocol: expected piece hash; context canceled; EOF Dec 31 01:06:20.978105122 (609h37m20.849188167s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978136983 (609h37m20.849220018s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection Dec 31 01:06:20.978148856 (609h37m20.849231901s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978171629 (609h37m20.849254674s) [seg_pos={0 0}, upload=534265]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:06:20.978256441 (609h37m20.849339476s) [upload=534265]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:06:20.978263184 (609h37m20.849346219s) [upload=534265]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:06:21.275269351 (609h37m21.146352406s) [upload=534265]: Done uploading segments... Dec 31 01:06:21.275294678 (609h37m21.146377713s) [upload=534265]: Upload finished. err: <nil> Duration: 32.43050698s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...XX.XXX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX.] duration: 6.115982137s trace: https://jaeger.dev.storj.io/trace/302875e9b5456820 FAIL (2025-12-31 01:10:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:09:30.952291082 (609h40m30.823374117s) [upload=534310]: Starting upload Dec 31 01:09:30.952342189 (609h40m30.823425234s) [upload=534310]: Done starting upload Dec 31 01:09:30.952511982 (609h40m30.823595017s) [upload=534310]: Uploading segments... Dec 31 01:09:30.952552770 (609h40m30.823635805s) [upload=534310]: Got next segment. Inline: false Dec 31 01:09:30.952579671 (609h40m30.823662706s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:09:30.952584220 (609h40m30.823667255s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:09:31.200060645 (609h40m31.07114369s) [seg_pos={0 0}, upload=534310]: Begin upload segment... Dec 31 01:09:31.200115720 (609h40m31.071198756s) [seg_pos={0 0}, upload=534310]: Done begin upload segment. Dec 31 01:09:31.200160485 (609h40m31.07124353s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.200183600 (609h40m31.071266635s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201328900 (609h40m31.072411945s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201655632 (609h40m31.072738678s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202001441 (609h40m31.073084476s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202298125 (609h40m31.07338116s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202753022 (609h40m31.073836067s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203062341 (609h40m31.074145386s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203432997 (609h40m31.074516032s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203781821 (609h40m31.074864866s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204750115 (609h40m31.07583315s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204981074 (609h40m31.076064119s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206060129 (609h40m31.077143164s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206134350 (609h40m31.077217385s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.207278809 (609h40m31.078361845s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.208923160 (609h40m31.080006205s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:32.904529566 (609h40m32.775612611s) [upload=534310]: Next returned nil segment Dec 31 01:09:32.904573228 (609h40m32.775656273s) [upload=534310]: Waiting for error group managing segments... Dec 31 01:09:33.177751323 (609h40m33.048834358s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.188115618 (609h40m33.059198663s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.193468411 (609h40m33.064551456s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.199886171 (609h40m33.070969206s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.203595614 (609h40m33.074678649s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.205080291 (609h40m33.076163326s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.209178975 (609h40m33.08026201s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.217865655 (609h40m33.0889487s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.219876184 (609h40m33.090959219s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.358860787 (609h40m33.229943832s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.383597680 (609h40m33.254680715s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.463911717 (609h40m33.334994762s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535677719 (609h40m33.406760754s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535698058 (609h40m33.406781103s) [seg_pos={0 0}, upload=534310]: Segment reached optimal threshold of 13 pieces. Dec 31 01:09:33.535862290 (609h40m33.406945335s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.535869223 (609h40m33.406952258s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.535958804 (609h40m33.407041849s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection Dec 31 01:09:33.535974513 (609h40m33.407057548s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536069374 (609h40m33.407152459s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.536085155 (609h40m33.40716819s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536110973 (609h40m33.407194018s) [seg_pos={0 0}, upload=534310]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:09:33.536258765 (609h40m33.407341811s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:09:33.536265258 (609h40m33.407348303s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:09:33.838979262 (609h40m33.710062308s) [upload=534310]: Done uploading segments... Dec 31 01:09:33.839001435 (609h40m33.71008448s) [upload=534310]: Upload finished. err: <nil> Duration: 6.115997127s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....XX.XXX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX] duration: 30.003219478s trace: https://jaeger.dev.storj.io/trace/302875e9b5456820 FAIL (2025-12-31 01:10:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:09:30.952291082 (609h40m30.823374117s) [upload=534310]: Starting upload Dec 31 01:09:30.952342189 (609h40m30.823425234s) [upload=534310]: Done starting upload Dec 31 01:09:30.952511982 (609h40m30.823595017s) [upload=534310]: Uploading segments... Dec 31 01:09:30.952552770 (609h40m30.823635805s) [upload=534310]: Got next segment. Inline: false Dec 31 01:09:30.952579671 (609h40m30.823662706s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:09:30.952584220 (609h40m30.823667255s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:09:31.200060645 (609h40m31.07114369s) [seg_pos={0 0}, upload=534310]: Begin upload segment... Dec 31 01:09:31.200115720 (609h40m31.071198756s) [seg_pos={0 0}, upload=534310]: Done begin upload segment. Dec 31 01:09:31.200160485 (609h40m31.07124353s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.200183600 (609h40m31.071266635s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201328900 (609h40m31.072411945s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201655632 (609h40m31.072738678s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202001441 (609h40m31.073084476s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202298125 (609h40m31.07338116s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202753022 (609h40m31.073836067s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203062341 (609h40m31.074145386s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203432997 (609h40m31.074516032s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203781821 (609h40m31.074864866s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204750115 (609h40m31.07583315s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204981074 (609h40m31.076064119s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206060129 (609h40m31.077143164s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206134350 (609h40m31.077217385s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.207278809 (609h40m31.078361845s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.208923160 (609h40m31.080006205s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:32.904529566 (609h40m32.775612611s) [upload=534310]: Next returned nil segment Dec 31 01:09:32.904573228 (609h40m32.775656273s) [upload=534310]: Waiting for error group managing segments... Dec 31 01:09:33.177751323 (609h40m33.048834358s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.188115618 (609h40m33.059198663s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.193468411 (609h40m33.064551456s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.199886171 (609h40m33.070969206s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.203595614 (609h40m33.074678649s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.205080291 (609h40m33.076163326s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.209178975 (609h40m33.08026201s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.217865655 (609h40m33.0889487s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.219876184 (609h40m33.090959219s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.358860787 (609h40m33.229943832s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.383597680 (609h40m33.254680715s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.463911717 (609h40m33.334994762s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535677719 (609h40m33.406760754s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535698058 (609h40m33.406781103s) [seg_pos={0 0}, upload=534310]: Segment reached optimal threshold of 13 pieces. Dec 31 01:09:33.535862290 (609h40m33.406945335s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.535869223 (609h40m33.406952258s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.535958804 (609h40m33.407041849s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection Dec 31 01:09:33.535974513 (609h40m33.407057548s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536069374 (609h40m33.407152459s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.536085155 (609h40m33.40716819s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536110973 (609h40m33.407194018s) [seg_pos={0 0}, upload=534310]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:09:33.536258765 (609h40m33.407341811s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:09:33.536265258 (609h40m33.407348303s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:09:33.838979262 (609h40m33.710062308s) [upload=534310]: Done uploading segments... Dec 31 01:09:33.839001435 (609h40m33.71008448s) [upload=534310]: Upload finished. err: <nil> Duration: 30.003229977s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X....XX.XXX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..X] duration: 30.003324163s trace: https://jaeger.dev.storj.io/trace/1dba1e705da16f9c FAIL (2025-12-31 01:09:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:09:00.701775153 (609h40m0.572858188s) [upload=534304]: Starting upload Dec 31 01:09:00.701899269 (609h40m0.572982314s) [upload=534304]: Done starting upload Dec 31 01:09:00.702186516 (609h40m0.573269561s) [upload=534304]: Uploading segments... Dec 31 01:09:00.702353704 (609h40m0.573436779s) [upload=534304]: Got next segment. Inline: false Dec 31 01:09:00.702412346 (609h40m0.573495391s) [upload=534304]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:09:00.702419129 (609h40m0.573502174s) [upload=534304]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:09:00.947352402 (609h40m0.818435457s) [seg_pos={0 0}, upload=534304]: Begin upload segment... Dec 31 01:09:00.947418848 (609h40m0.818501893s) [seg_pos={0 0}, upload=534304]: Done begin upload segment. Dec 31 01:09:00.947479895 (609h40m0.81856294s) [noise=true , address=145.79.227.48:4302 , piece_id=ZYLDEARMSOT5PMK5MJUSK24U7VCLNMCSU7I335TI44JNO5GJ7FIA, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.947561891 (609h40m0.818644946s) [noise=true , address=77.90.31.19:4844 , piece_id=UJGR7CWYSZ6HJQ7VS5CENXRI5TBYDLNDXBZAT2DE4HM55L77HFCQ, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.948468197 (609h40m0.819551242s) [noise=true , address=145.79.227.31:4797 , piece_id=KGYBJKJT56RUZ7MZLPRRKPPMNIYQTNA2YO55NOFDLKTZ2NFI4DCA, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.949348273 (609h40m0.820431308s) [noise=true , address=77.90.31.12:4589 , piece_id=YKFPWHTDJ3O3PAN7XK34VTJOWD5TCNHPQ5643PNGDD7XZH365ETA, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.950004232 (609h40m0.821087277s) [noise=true , address=77.90.31.14:4664 , piece_id=J5HDVR7CSBJ65Q454MJXEX4UPOONL2BZZHHA3SBU3PHZCAQGNU5A, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.950456954 (609h40m0.821539999s) [noise=true , address=145.79.227.24:4582 , piece_id=LWGFFVCFW7TPOICNXNCRMQIPH2FQA3NGHQSHZ7IKQTFTTTRDS54A, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.950673286 (609h40m0.821756331s) [noise=true , address=45.91.116.13:4155 , piece_id=UN3M4M3FLQCGHN3D2CZT7FCM3TIGB5L5DIF5RF3CNWULSRXGIP2A, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.951114737 (609h40m0.822197782s) [noise=true , address=45.91.116.12:4228 , piece_id=KCEFSSOX2S3A6NJHLGQKUCH4UBY5ZZYA6PHKD35NCXG2BRJTRISA, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.951377396 (609h40m0.822460441s) [noise=true , address=145.79.227.50:4374 , piece_id=OIINHZPTEUD2ZJCZYJLVLQAWVFL3IBECGSDFUILCGZ5BMURY7ZGQ, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.951559633 (609h40m0.822642678s) [noise=true , address=77.90.31.17:4773 , piece_id=WTWJWN7OOHIM3DQSPYDWDBQL3E6PV6HSR6Y33V77JP3CPF3S24CQ, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.951866758 (609h40m0.822949793s) [noise=true , address=145.79.227.26:4655 , piece_id=S7RPRBQO2567YXI25C6VY33CBI7FV2J4ZSL7WLVKKGU35JBTYUMQ, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.958098214 (609h40m0.829181249s) [noise=true , address=45.91.116.6:4976 , piece_id=LCKEICKRRY2NTH2W62SS343SM6WSIEEXMENKDJ5RFCSF5NFOKWWA, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.960822531 (609h40m0.831905576s) [noise=true , address=45.91.116.9:4262 , piece_id=IKUOOQVZHMDUVQX246PM35QHUXTFYO2C762D2MK5XCC3O4CRTFBA, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.962623200 (609h40m0.833706245s) [noise=true , address=145.79.227.47:4267 , piece_id=MRMAD622MJREQJEGJGSY3ESHFWBE5EHT72NVI445UCIB75UODCNQ, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.964541893 (609h40m0.835624938s) [noise=true , address=145.79.227.45:4231 , piece_id=VMFZDV6OMG2S6DYFEZR2UDDONQ6HW6FKVNP3XPWSZ7PFW3KGBIRA, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.966244204 (609h40m0.837327249s) [noise=true , address=145.79.227.11:4730 , piece_id=4DBOUI5NKMXEQLIW6LVJKVZNLCYBKWTJMZHPMGIGI5BWLD44P2EA, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:02.664885536 (609h40m2.535968581s) [upload=534304]: Next returned nil segment Dec 31 01:09:02.664905624 (609h40m2.535988669s) [upload=534304]: Waiting for error group managing segments... Dec 31 01:09:03.015637030 (609h40m2.886720095s) [noise=true , address=145.79.227.50:4374 , piece_id=OIINHZPTEUD2ZJCZYJLVLQAWVFL3IBECGSDFUILCGZ5BMURY7ZGQ, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.016352122 (609h40m2.887435167s) [noise=true , address=145.79.227.48:4302 , piece_id=ZYLDEARMSOT5PMK5MJUSK24U7VCLNMCSU7I335TI44JNO5GJ7FIA, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.016523729 (609h40m2.887606804s) [noise=true , address=145.79.227.24:4582 , piece_id=LWGFFVCFW7TPOICNXNCRMQIPH2FQA3NGHQSHZ7IKQTFTTTRDS54A, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.050886209 (609h40m2.921969254s) [noise=true , address=145.79.227.45:4231 , piece_id=VMFZDV6OMG2S6DYFEZR2UDDONQ6HW6FKVNP3XPWSZ7PFW3KGBIRA, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.054784321 (609h40m2.925867366s) [noise=true , address=145.79.227.31:4797 , piece_id=KGYBJKJT56RUZ7MZLPRRKPPMNIYQTNA2YO55NOFDLKTZ2NFI4DCA, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.097354450 (609h40m2.968437515s) [noise=true , address=145.79.227.11:4730 , piece_id=4DBOUI5NKMXEQLIW6LVJKVZNLCYBKWTJMZHPMGIGI5BWLD44P2EA, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.115899222 (609h40m2.986982277s) [noise=true , address=145.79.227.26:4655 , piece_id=S7RPRBQO2567YXI25C6VY33CBI7FV2J4ZSL7WLVKKGU35JBTYUMQ, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.119796834 (609h40m2.990879889s) [noise=true , address=145.79.227.47:4267 , piece_id=MRMAD622MJREQJEGJGSY3ESHFWBE5EHT72NVI445UCIB75UODCNQ, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.156087084 (609h40m3.027170119s) [noise=true , address=77.90.31.14:4664 , piece_id=J5HDVR7CSBJ65Q454MJXEX4UPOONL2BZZHHA3SBU3PHZCAQGNU5A, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.161484001 (609h40m3.032567046s) [noise=true , address=77.90.31.19:4844 , piece_id=UJGR7CWYSZ6HJQ7VS5CENXRI5TBYDLNDXBZAT2DE4HM55L77HFCQ, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.169528880 (609h40m3.040611925s) [noise=true , address=77.90.31.17:4773 , piece_id=WTWJWN7OOHIM3DQSPYDWDBQL3E6PV6HSR6Y33V77JP3CPF3S24CQ, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.353124388 (609h40m3.224207433s) [noise=true , address=45.91.116.9:4262 , piece_id=IKUOOQVZHMDUVQX246PM35QHUXTFYO2C762D2MK5XCC3O4CRTFBA, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.408731902 (609h40m3.279814947s) [noise=true , address=45.91.116.12:4228 , piece_id=KCEFSSOX2S3A6NJHLGQKUCH4UBY5ZZYA6PHKD35NCXG2BRJTRISA, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.408759785 (609h40m3.27984283s) [seg_pos={0 0}, upload=534304]: Segment reached optimal threshold of 13 pieces. Dec 31 01:09:03.409102829 (609h40m3.280185884s) [noise=true , address=45.91.116.13:4155 , piece_id=UN3M4M3FLQCGHN3D2CZT7FCM3TIGB5L5DIF5RF3CNWULSRXGIP2A, seg_pos={0 0}, upload=534304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:03.409113158 (609h40m3.280196253s) [noise=true , address=45.91.116.13:4155 , piece_id=UN3M4M3FLQCGHN3D2CZT7FCM3TIGB5L5DIF5RF3CNWULSRXGIP2A, seg_pos={0 0}, upload=534304]: Long tail cancellation due to optimal threshold Dec 31 01:09:03.409669688 (609h40m3.280752733s) [noise=true , address=77.90.31.12:4589 , piece_id=YKFPWHTDJ3O3PAN7XK34VTJOWD5TCNHPQ5643PNGDD7XZH365ETA, seg_pos={0 0}, upload=534304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:56712->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:56712->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56712->77.90.31.12:4589: use of closed network connection Dec 31 01:09:03.409685067 (609h40m3.280768112s) [noise=true , address=77.90.31.12:4589 , piece_id=YKFPWHTDJ3O3PAN7XK34VTJOWD5TCNHPQ5643PNGDD7XZH365ETA, seg_pos={0 0}, upload=534304]: Long tail cancellation due to optimal threshold Dec 31 01:09:03.409964418 (609h40m3.281047484s) [noise=true , address=45.91.116.6:4976 , piece_id=LCKEICKRRY2NTH2W62SS343SM6WSIEEXMENKDJ5RFCSF5NFOKWWA, seg_pos={0 0}, upload=534304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:03.409975440 (609h40m3.281058495s) [noise=true , address=45.91.116.6:4976 , piece_id=LCKEICKRRY2NTH2W62SS343SM6WSIEEXMENKDJ5RFCSF5NFOKWWA, seg_pos={0 0}, upload=534304]: Long tail cancellation due to optimal threshold Dec 31 01:09:03.409998093 (609h40m3.281081138s) [seg_pos={0 0}, upload=534304]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:09:03.410976656 (609h40m3.282059711s) [upload=534304]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:09:03.410987187 (609h40m3.282070232s) [upload=534304]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:09:03.708184178 (609h40m3.579267904s) [upload=534304]: Done uploading segments... Dec 31 01:09:03.708235224 (609h40m3.579318259s) [upload=534304]: Upload finished. err: <nil> Duration: 30.00335387s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.470140396s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:09:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.470163461s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 4.849477033s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:09:17 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 4.849502662s core uplink eu1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)