Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [..X......X..........X.............X...............X...X.....] duration:
55.397726871s
FAIL (2025-07-09 02:17:55 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
.
Jul 9 02:12:07.627735816 (377h1m42.398377151s) [upload=301626]: Got next segment. Inline: false
Jul 9 02:12:07.627768629 (377h1m42.398409964s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:12:07.627773568 (377h1m42.398414903s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:12:08.075105689 (377h1m42.845747024s) [seg_pos={0 0}, upload=301626]: Begin upload segment...
Jul 9 02:12:08.075172738 (377h1m42.845814073s) [seg_pos={0 0}, upload=301626]: Done begin upload segment.
Jul 9 02:12:08.075200341 (377h1m42.845841676s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075226711 (377h1m42.845868046s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075642747 (377h1m42.846284102s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076099570 (377h1m42.846740905s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076193791 (377h1m42.846835126s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076453428 (377h1m42.847094763s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076686334 (377h1m42.847327669s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076926883 (377h1m42.847568218s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076956250 (377h1m42.847597585s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077181110 (377h1m42.847822445s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077198733 (377h1m42.847840068s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077433213 (377h1m42.848074548s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077452068 (377h1m42.848093403s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077655618 (377h1m42.848296953s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077884866 (377h1m42.848526201s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078100028 (377h1m42.848741363s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078310551 (377h1m42.848951886s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078506095 (377h1m42.84914743s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:09.631539904 (377h1m44.402181239s) [upload=301626]: Next returned nil segment
Jul 9 02:12:09.631556275 (377h1m44.4021976s) [upload=301626]: Waiting for error group managing segments...
Jul 9 02:12:10.026556908 (377h1m44.797198243s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.029581435 (377h1m44.80022277s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.030348031 (377h1m44.800989366s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.037300182 (377h1m44.807941517s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.038590761 (377h1m44.809232096s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.040614022 (377h1m44.811255347s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.243165042 (377h1m45.013806377s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.294115297 (377h1m45.064756632s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:12.752127764 (377h1m47.522769099s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:29.819338082 (377h2m4.589979417s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:35.349456003 (377h2m10.120097338s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:40.626537094 (377h2m15.397178419s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:42.766847833 (377h2m17.537489168s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:13:06.981429964 (377h2m41.752071299s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:15mgZMkU28J1JwrRd4XbvZFyjp8G7CTBkJCxbzUYqKeSeqCZ9F, address:104.36.82.7:4944): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981472496 (377h2m41.752113821s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; EOF
Jul 9 02:13:06.981498496 (377h2m41.752139821s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:12pHU3Xp1sDgnmYMcbWZD2DivxdEUzqnBVJaiRaQipuRy2d75u5, address:104.36.82.18:4375): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981567808 (377h2m41.752209143s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981595070 (377h2m41.752236395s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection
Jul 9 02:13:06.981611983 (377h2m41.752253318s) [seg_pos={0 0}, upload=301626]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:13:07.214001409 (377h2m41.984642744s) [upload=301626]: Done uploading segments...
Jul 9 02:13:07.214031306 (377h2m41.984672641s) [upload=301626]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 55.397748401s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [............X......X..X...X.X.X.X...X...............X.....X.] duration:
28.910269229s
FAIL (2025-07-09 02:17:53 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 28.910287706s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [................X.................X......X.........XXX.X....] duration:
5.979336771s
FAIL (2025-07-09 02:17:25 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
tchRequestItem_ObjectCommit
Jul 9 02:13:15.533673746 (377h2m50.304315071s) [upload=301640]: Done uploading segments...
Jul 9 02:13:15.533714053 (377h2m50.304355388s) [upload=301640]: Upload finished. err: <nil>
Jul 9 02:14:11.239664264 (377h3m46.010305599s) [upload=301655]: Starting upload
Jul 9 02:14:11.239720071 (377h3m46.010361406s) [upload=301655]: Done starting upload
Jul 9 02:14:11.239901257 (377h3m46.010542592s) [upload=301655]: Uploading segments...
Jul 9 02:14:11.239936835 (377h3m46.01057816s) [upload=301655]: Got next segment. Inline: false
Jul 9 02:14:11.239961161 (377h3m46.010602496s) [upload=301655]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:14:11.239965630 (377h3m46.010606965s) [upload=301655]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:14:11.478079609 (377h3m46.248720944s) [seg_pos={0 0}, upload=301655]: Begin upload segment...
Jul 9 02:14:11.478117833 (377h3m46.248759158s) [seg_pos={0 0}, upload=301655]: Done begin upload segment.
Jul 9 02:14:11.478140505 (377h3m46.24878184s) [noise=true , address=45.91.116.13:4155 , piece_id=D7BUOVLC3Q6T5GX67ZSA6NZNI4RTDVMULDK32D3EAQ2GWGEUL6FQ, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.478246749 (377h3m46.248888084s) [noise=true , address=104.36.82.6:4876 , piece_id=XQUCW55LNBVJBY2I73LBSVISZ54JFW4KSSD3SSI4C2QMWEPXSX4Q, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.478579806 (377h3m46.249221141s) [noise=true , address=104.36.82.6:4874 , piece_id=PKM52XEQYB62IF75NYSPLQZNHRMO7QE56QLIHPC7ZIJBRJ3HOVBA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.478878968 (377h3m46.249520303s) [noise=true , address=45.91.116.4:4939 , piece_id=OMGUKZV52SVS2MK4LMVTSJGIEKCNTS7QU36FCXQTN3JIXNWTGN4A, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479123196 (377h3m46.249764531s) [noise=true , address=104.36.82.12:4149 , piece_id=K36PH6SAQRP2EOSPXELGH7KLA3PTLG77JGL5YVCELPFY7SLHLQ5A, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479199021 (377h3m46.249840376s) [noise=true , address=45.91.116.12:4226 , piece_id=VZ7UJ3JWBCXUKX5VAQVOK2KKN6T4BJX2TMHRMN5FUERR5IMN7ZWA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479511829 (377h3m46.250153164s) [noise=true , address=45.91.116.8:4045 , piece_id=MCNAFQZCMOL57CK6WJIBV2HIL65GD3FW7G5PIVKFVBXXUXYFX6LA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479780483 (377h3m46.250421819s) [noise=true , address=104.36.82.8:4970 , piece_id=E6OKLTVW3JMOQOJZUKVC4GBTLSW7CZ2NBVFGO4UUJQ66JBUDE7RA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479798347 (377h3m46.250439672s) [noise=true , address=104.36.82.5:4840 , piece_id=CCMPPQEVRQA2X2LXJ4PSM5ZH2IABEF45G66Q27NV6E4NCSVDOL2A, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480259009 (377h3m46.250900334s) [noise=true , address=104.36.82.16:4294 , piece_id=QVJIAM6GMAJKETPJQPACPSQCR5ITZH5ILHJJWF6LXVEZBNTX56JQ, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480355454 (377h3m46.250996789s) [noise=true , address=77.90.31.11:4553 , piece_id=L7ABAZT6JX265EIDHNPXQXTP3RMXRTPPCMCD76NJTPMBMTCKFIMA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480653393 (377h3m46.251294738s) [noise=true , address=77.90.31.21:4879 , piece_id=LP5SORULTBNCFPGOMLPLDJQ3WTHBXHTJ3MLFXZFW65FW6ITY2BTA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480924812 (377h3m46.251566147s) [noise=true , address=45.91.116.8:4047 , piece_id=JQV6KR6HJDYNN3AASSY2K4B3T673XSXKUYRT6LR3TCB2N43G54ZA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480949319 (377h3m46.251590654s) [noise=true , address=77.90.31.10:4518 , piece_id=ZWOI3W26Y5MBWHP3PIVHYQ67LDLDPQ5TWPPXDJS7LEYW4EVVCORA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.481249524 (377h3m46.251890859s) [noise=true , address=45.91.116.14:4190 , piece_id=MOKLQKDHNXPZGY7FVTH6PQ7QXOEEWMIDBEMLUOPNWJYV4H3WTP6Q, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.481495965 (377h3m46.2521373s) [noise=true , address=77.90.31.21:4878 , piece_id=D7W27EYYNVA7Z5U2XC66PZZLSME3GQZOBFXQKADR2QE7WASPOROQ, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:13.042937288 (377h3m47.813578663s) [upload=301655]: Next returned nil segment
Jul 9 02:14:13.042961194 (377h3m47.813602549s) [upload=301655]: Waiting for error group managing segments...
Jul 9 02:14:13.363736407 (377h3m48.134377742s) [noise=true , address=104.36.82.5:4840 , piece_id=CCMPPQEVRQA2X2LXJ4PSM5ZH2IABEF45G66Q27NV6E4NCSVDOL2A, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.539916948 (377h3m48.310558283s) [noise=true , address=45.91.116.13:4155 , piece_id=D7BUOVLC3Q6T5GX67ZSA6NZNI4RTDVMULDK32D3EAQ2GWGEUL6FQ, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.542285079 (377h3m48.312926414s) [noise=true , address=45.91.116.14:4190 , piece_id=MOKLQKDHNXPZGY7FVTH6PQ7QXOEEWMIDBEMLUOPNWJYV4H3WTP6Q, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.547444790 (377h3m48.318086135s) [noise=true , address=45.91.116.12:4226 , piece_id=VZ7UJ3JWBCXUKX5VAQVOK2KKN6T4BJX2TMHRMN5FUERR5IMN7ZWA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.556717610 (377h3m48.327358945s) [noise=true , address=104.36.82.16:4294 , piece_id=QVJIAM6GMAJKETPJQPACPSQCR5ITZH5ILHJJWF6LXVEZBNTX56JQ, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.565064359 (377h3m48.335705694s) [noise=true , address=77.90.31.21:4879 , piece_id=LP5SORULTBNCFPGOMLPLDJQ3WTHBXHTJ3MLFXZFW65FW6ITY2BTA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.568914063 (377h3m48.339555388s) [noise=true , address=77.90.31.10:4518 , piece_id=ZWOI3W26Y5MBWHP3PIVHYQ67LDLDPQ5TWPPXDJS7LEYW4EVVCORA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.571640700 (377h3m48.342282035s) [noise=true , address=77.90.31.11:4553 , piece_id=L7ABAZT6JX265EIDHNPXQXTP3RMXRTPPCMCD76NJTPMBMTCKFIMA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.576586953 (377h3m48.347228288s) [noise=true , address=77.90.31.21:4878 , piece_id=D7W27EYYNVA7Z5U2XC66PZZLSME3GQZOBFXQKADR2QE7WASPOROQ, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.607484504 (377h3m48.378125839s) [noise=true , address=45.91.116.4:4939 , piece_id=OMGUKZV52SVS2MK4LMVTSJGIEKCNTS7QU36FCXQTN3JIXNWTGN4A, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.618215825 (377h3m48.38885716s) [noise=true , address=45.91.116.8:4047 , piece_id=JQV6KR6HJDYNN3AASSY2K4B3T673XSXKUYRT6LR3TCB2N43G54ZA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.695873867 (377h3m48.466515202s) [noise=true , address=45.91.116.8:4045 , piece_id=MCNAFQZCMOL57CK6WJIBV2HIL65GD3FW7G5PIVKFVBXXUXYFX6LA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:40.587881399 (377h4m15.358522734s) [noise=true , address=104.36.82.8:4970 , piece_id=E6OKLTVW3JMOQOJZUKVC4GBTLSW7CZ2NBVFGO4UUJQ66JBUDE7RA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 188.166.225.170:47376->104.36.82.8:4970: use of closed network connection; write tcp 188.166.225.170:47376->104.36.82.8:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47376->104.36.82.8:4970: use of closed network connection
Jul 9 02:14:40.587919511 (377h4m15.358560846s) [noise=true , address=104.36.82.12:4149 , piece_id=K36PH6SAQRP2EOSPXELGH7KLA3PTLG77JGL5YVCELPFY7SLHLQ5A, seg_pos={0 0}, upload=301655]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:51768->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:51768->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->104.36.82.12:4149: use of closed network connection
Jul 9 02:14:40.587938768 (377h4m15.358580103s) [noise=true , address=104.36.82.6:4874 , piece_id=PKM52XEQYB62IF75NYSPLQZNHRMO7QE56QLIHPC7ZIJBRJ3HOVBA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:56922->104.36.82.6:4874: use of closed network connection; write tcp 188.166.225.170:56922->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56922->104.36.82.6:4874: use of closed network connection
Jul 9 02:14:40.587999454 (377h4m15.358640779s) [noise=true , address=104.36.82.6:4876 , piece_id=XQUCW55LNBVJBY2I73LBSVISZ54JFW4KSSD3SSI4C2QMWEPXSX4Q, seg_pos={0 0}, upload=301655]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:41436->104.36.82.6:4876: use of closed network connection; write tcp 188.166.225.170:41436->104.36.82.6:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41436->104.36.82.6:4876: use of closed network connection
Jul 9 02:14:40.588019132 (377h4m15.358660457s) [seg_pos={0 0}, upload=301655]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:14:40.820632878 (377h4m15.591274213s) [upload=301655]: Done uploading segments...
Jul 9 02:14:40.820657295 (377h4m15.59129862s) [upload=301655]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.979374343s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
17 out of 60 is failed (threshold=3) [.XX..X......XXX....XX.....XX...X.XX........X.X......X..X....] duration:
31.037904469s
FAIL (2025-07-09 02:17:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 31.037918875s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [X............X......X..X...X.X.X.X...X...............X.....X] duration:
30.001173722s
FAIL (2025-07-09 02:17:01 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001193169s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [X.XX..X......XXX....XX.....XX...X.XX........X.X......X..X...] duration:
25.689965576s
FAIL (2025-07-09 02:16:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 25.689982377s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [...X......X..........X.............X...............X...X....] duration:
25.367236226s
FAIL (2025-07-09 02:16:38 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
.
Jul 9 02:12:07.627735816 (377h1m42.398377151s) [upload=301626]: Got next segment. Inline: false
Jul 9 02:12:07.627768629 (377h1m42.398409964s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:12:07.627773568 (377h1m42.398414903s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:12:08.075105689 (377h1m42.845747024s) [seg_pos={0 0}, upload=301626]: Begin upload segment...
Jul 9 02:12:08.075172738 (377h1m42.845814073s) [seg_pos={0 0}, upload=301626]: Done begin upload segment.
Jul 9 02:12:08.075200341 (377h1m42.845841676s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075226711 (377h1m42.845868046s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075642747 (377h1m42.846284102s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076099570 (377h1m42.846740905s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076193791 (377h1m42.846835126s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076453428 (377h1m42.847094763s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076686334 (377h1m42.847327669s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076926883 (377h1m42.847568218s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076956250 (377h1m42.847597585s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077181110 (377h1m42.847822445s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077198733 (377h1m42.847840068s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077433213 (377h1m42.848074548s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077452068 (377h1m42.848093403s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077655618 (377h1m42.848296953s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077884866 (377h1m42.848526201s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078100028 (377h1m42.848741363s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078310551 (377h1m42.848951886s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078506095 (377h1m42.84914743s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:09.631539904 (377h1m44.402181239s) [upload=301626]: Next returned nil segment
Jul 9 02:12:09.631556275 (377h1m44.4021976s) [upload=301626]: Waiting for error group managing segments...
Jul 9 02:12:10.026556908 (377h1m44.797198243s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.029581435 (377h1m44.80022277s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.030348031 (377h1m44.800989366s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.037300182 (377h1m44.807941517s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.038590761 (377h1m44.809232096s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.040614022 (377h1m44.811255347s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.243165042 (377h1m45.013806377s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.294115297 (377h1m45.064756632s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:12.752127764 (377h1m47.522769099s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:29.819338082 (377h2m4.589979417s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:35.349456003 (377h2m10.120097338s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:40.626537094 (377h2m15.397178419s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:42.766847833 (377h2m17.537489168s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:13:06.981429964 (377h2m41.752071299s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:15mgZMkU28J1JwrRd4XbvZFyjp8G7CTBkJCxbzUYqKeSeqCZ9F, address:104.36.82.7:4944): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981472496 (377h2m41.752113821s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; EOF
Jul 9 02:13:06.981498496 (377h2m41.752139821s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:12pHU3Xp1sDgnmYMcbWZD2DivxdEUzqnBVJaiRaQipuRy2d75u5, address:104.36.82.18:4375): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981567808 (377h2m41.752209143s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981595070 (377h2m41.752236395s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection
Jul 9 02:13:06.981611983 (377h2m41.752253318s) [seg_pos={0 0}, upload=301626]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:13:07.214001409 (377h2m41.984642744s) [upload=301626]: Done uploading segments...
Jul 9 02:13:07.214031306 (377h2m41.984672641s) [upload=301626]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 25.367267005s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [.................X.................X......X.........XXX.X...] duration:
5.767187234s
FAIL (2025-07-09 02:16:19 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
tchRequestItem_ObjectCommit
Jul 9 02:13:15.533673746 (377h2m50.304315071s) [upload=301640]: Done uploading segments...
Jul 9 02:13:15.533714053 (377h2m50.304355388s) [upload=301640]: Upload finished. err: <nil>
Jul 9 02:14:11.239664264 (377h3m46.010305599s) [upload=301655]: Starting upload
Jul 9 02:14:11.239720071 (377h3m46.010361406s) [upload=301655]: Done starting upload
Jul 9 02:14:11.239901257 (377h3m46.010542592s) [upload=301655]: Uploading segments...
Jul 9 02:14:11.239936835 (377h3m46.01057816s) [upload=301655]: Got next segment. Inline: false
Jul 9 02:14:11.239961161 (377h3m46.010602496s) [upload=301655]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:14:11.239965630 (377h3m46.010606965s) [upload=301655]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:14:11.478079609 (377h3m46.248720944s) [seg_pos={0 0}, upload=301655]: Begin upload segment...
Jul 9 02:14:11.478117833 (377h3m46.248759158s) [seg_pos={0 0}, upload=301655]: Done begin upload segment.
Jul 9 02:14:11.478140505 (377h3m46.24878184s) [noise=true , address=45.91.116.13:4155 , piece_id=D7BUOVLC3Q6T5GX67ZSA6NZNI4RTDVMULDK32D3EAQ2GWGEUL6FQ, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.478246749 (377h3m46.248888084s) [noise=true , address=104.36.82.6:4876 , piece_id=XQUCW55LNBVJBY2I73LBSVISZ54JFW4KSSD3SSI4C2QMWEPXSX4Q, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.478579806 (377h3m46.249221141s) [noise=true , address=104.36.82.6:4874 , piece_id=PKM52XEQYB62IF75NYSPLQZNHRMO7QE56QLIHPC7ZIJBRJ3HOVBA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.478878968 (377h3m46.249520303s) [noise=true , address=45.91.116.4:4939 , piece_id=OMGUKZV52SVS2MK4LMVTSJGIEKCNTS7QU36FCXQTN3JIXNWTGN4A, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479123196 (377h3m46.249764531s) [noise=true , address=104.36.82.12:4149 , piece_id=K36PH6SAQRP2EOSPXELGH7KLA3PTLG77JGL5YVCELPFY7SLHLQ5A, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479199021 (377h3m46.249840376s) [noise=true , address=45.91.116.12:4226 , piece_id=VZ7UJ3JWBCXUKX5VAQVOK2KKN6T4BJX2TMHRMN5FUERR5IMN7ZWA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479511829 (377h3m46.250153164s) [noise=true , address=45.91.116.8:4045 , piece_id=MCNAFQZCMOL57CK6WJIBV2HIL65GD3FW7G5PIVKFVBXXUXYFX6LA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479780483 (377h3m46.250421819s) [noise=true , address=104.36.82.8:4970 , piece_id=E6OKLTVW3JMOQOJZUKVC4GBTLSW7CZ2NBVFGO4UUJQ66JBUDE7RA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479798347 (377h3m46.250439672s) [noise=true , address=104.36.82.5:4840 , piece_id=CCMPPQEVRQA2X2LXJ4PSM5ZH2IABEF45G66Q27NV6E4NCSVDOL2A, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480259009 (377h3m46.250900334s) [noise=true , address=104.36.82.16:4294 , piece_id=QVJIAM6GMAJKETPJQPACPSQCR5ITZH5ILHJJWF6LXVEZBNTX56JQ, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480355454 (377h3m46.250996789s) [noise=true , address=77.90.31.11:4553 , piece_id=L7ABAZT6JX265EIDHNPXQXTP3RMXRTPPCMCD76NJTPMBMTCKFIMA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480653393 (377h3m46.251294738s) [noise=true , address=77.90.31.21:4879 , piece_id=LP5SORULTBNCFPGOMLPLDJQ3WTHBXHTJ3MLFXZFW65FW6ITY2BTA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480924812 (377h3m46.251566147s) [noise=true , address=45.91.116.8:4047 , piece_id=JQV6KR6HJDYNN3AASSY2K4B3T673XSXKUYRT6LR3TCB2N43G54ZA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480949319 (377h3m46.251590654s) [noise=true , address=77.90.31.10:4518 , piece_id=ZWOI3W26Y5MBWHP3PIVHYQ67LDLDPQ5TWPPXDJS7LEYW4EVVCORA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.481249524 (377h3m46.251890859s) [noise=true , address=45.91.116.14:4190 , piece_id=MOKLQKDHNXPZGY7FVTH6PQ7QXOEEWMIDBEMLUOPNWJYV4H3WTP6Q, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.481495965 (377h3m46.2521373s) [noise=true , address=77.90.31.21:4878 , piece_id=D7W27EYYNVA7Z5U2XC66PZZLSME3GQZOBFXQKADR2QE7WASPOROQ, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:13.042937288 (377h3m47.813578663s) [upload=301655]: Next returned nil segment
Jul 9 02:14:13.042961194 (377h3m47.813602549s) [upload=301655]: Waiting for error group managing segments...
Jul 9 02:14:13.363736407 (377h3m48.134377742s) [noise=true , address=104.36.82.5:4840 , piece_id=CCMPPQEVRQA2X2LXJ4PSM5ZH2IABEF45G66Q27NV6E4NCSVDOL2A, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.539916948 (377h3m48.310558283s) [noise=true , address=45.91.116.13:4155 , piece_id=D7BUOVLC3Q6T5GX67ZSA6NZNI4RTDVMULDK32D3EAQ2GWGEUL6FQ, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.542285079 (377h3m48.312926414s) [noise=true , address=45.91.116.14:4190 , piece_id=MOKLQKDHNXPZGY7FVTH6PQ7QXOEEWMIDBEMLUOPNWJYV4H3WTP6Q, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.547444790 (377h3m48.318086135s) [noise=true , address=45.91.116.12:4226 , piece_id=VZ7UJ3JWBCXUKX5VAQVOK2KKN6T4BJX2TMHRMN5FUERR5IMN7ZWA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.556717610 (377h3m48.327358945s) [noise=true , address=104.36.82.16:4294 , piece_id=QVJIAM6GMAJKETPJQPACPSQCR5ITZH5ILHJJWF6LXVEZBNTX56JQ, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.565064359 (377h3m48.335705694s) [noise=true , address=77.90.31.21:4879 , piece_id=LP5SORULTBNCFPGOMLPLDJQ3WTHBXHTJ3MLFXZFW65FW6ITY2BTA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.568914063 (377h3m48.339555388s) [noise=true , address=77.90.31.10:4518 , piece_id=ZWOI3W26Y5MBWHP3PIVHYQ67LDLDPQ5TWPPXDJS7LEYW4EVVCORA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.571640700 (377h3m48.342282035s) [noise=true , address=77.90.31.11:4553 , piece_id=L7ABAZT6JX265EIDHNPXQXTP3RMXRTPPCMCD76NJTPMBMTCKFIMA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.576586953 (377h3m48.347228288s) [noise=true , address=77.90.31.21:4878 , piece_id=D7W27EYYNVA7Z5U2XC66PZZLSME3GQZOBFXQKADR2QE7WASPOROQ, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.607484504 (377h3m48.378125839s) [noise=true , address=45.91.116.4:4939 , piece_id=OMGUKZV52SVS2MK4LMVTSJGIEKCNTS7QU36FCXQTN3JIXNWTGN4A, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.618215825 (377h3m48.38885716s) [noise=true , address=45.91.116.8:4047 , piece_id=JQV6KR6HJDYNN3AASSY2K4B3T673XSXKUYRT6LR3TCB2N43G54ZA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.695873867 (377h3m48.466515202s) [noise=true , address=45.91.116.8:4045 , piece_id=MCNAFQZCMOL57CK6WJIBV2HIL65GD3FW7G5PIVKFVBXXUXYFX6LA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:40.587881399 (377h4m15.358522734s) [noise=true , address=104.36.82.8:4970 , piece_id=E6OKLTVW3JMOQOJZUKVC4GBTLSW7CZ2NBVFGO4UUJQ66JBUDE7RA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 188.166.225.170:47376->104.36.82.8:4970: use of closed network connection; write tcp 188.166.225.170:47376->104.36.82.8:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47376->104.36.82.8:4970: use of closed network connection
Jul 9 02:14:40.587919511 (377h4m15.358560846s) [noise=true , address=104.36.82.12:4149 , piece_id=K36PH6SAQRP2EOSPXELGH7KLA3PTLG77JGL5YVCELPFY7SLHLQ5A, seg_pos={0 0}, upload=301655]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:51768->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:51768->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->104.36.82.12:4149: use of closed network connection
Jul 9 02:14:40.587938768 (377h4m15.358580103s) [noise=true , address=104.36.82.6:4874 , piece_id=PKM52XEQYB62IF75NYSPLQZNHRMO7QE56QLIHPC7ZIJBRJ3HOVBA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:56922->104.36.82.6:4874: use of closed network connection; write tcp 188.166.225.170:56922->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56922->104.36.82.6:4874: use of closed network connection
Jul 9 02:14:40.587999454 (377h4m15.358640779s) [noise=true , address=104.36.82.6:4876 , piece_id=XQUCW55LNBVJBY2I73LBSVISZ54JFW4KSSD3SSI4C2QMWEPXSX4Q, seg_pos={0 0}, upload=301655]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:41436->104.36.82.6:4876: use of closed network connection; write tcp 188.166.225.170:41436->104.36.82.6:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41436->104.36.82.6:4876: use of closed network connection
Jul 9 02:14:40.588019132 (377h4m15.358660457s) [seg_pos={0 0}, upload=301655]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:14:40.820632878 (377h4m15.591274213s) [upload=301655]: Done uploading segments...
Jul 9 02:14:40.820657295 (377h4m15.59129862s) [upload=301655]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.767207672s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
19 out of 60 is failed (threshold=3) [XX.XX..X......XXX....XX.....XX...X.XX........X.X......X..X..] duration:
44.646138308s
FAIL (2025-07-09 02:15:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 44.646187612s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [..................X.................X......X.........XXX.X..] duration:
5.797955587s
FAIL (2025-07-09 02:15:39 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
tchRequestItem_ObjectCommit
Jul 9 02:13:15.533673746 (377h2m50.304315071s) [upload=301640]: Done uploading segments...
Jul 9 02:13:15.533714053 (377h2m50.304355388s) [upload=301640]: Upload finished. err: <nil>
Jul 9 02:14:11.239664264 (377h3m46.010305599s) [upload=301655]: Starting upload
Jul 9 02:14:11.239720071 (377h3m46.010361406s) [upload=301655]: Done starting upload
Jul 9 02:14:11.239901257 (377h3m46.010542592s) [upload=301655]: Uploading segments...
Jul 9 02:14:11.239936835 (377h3m46.01057816s) [upload=301655]: Got next segment. Inline: false
Jul 9 02:14:11.239961161 (377h3m46.010602496s) [upload=301655]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:14:11.239965630 (377h3m46.010606965s) [upload=301655]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:14:11.478079609 (377h3m46.248720944s) [seg_pos={0 0}, upload=301655]: Begin upload segment...
Jul 9 02:14:11.478117833 (377h3m46.248759158s) [seg_pos={0 0}, upload=301655]: Done begin upload segment.
Jul 9 02:14:11.478140505 (377h3m46.24878184s) [noise=true , address=45.91.116.13:4155 , piece_id=D7BUOVLC3Q6T5GX67ZSA6NZNI4RTDVMULDK32D3EAQ2GWGEUL6FQ, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.478246749 (377h3m46.248888084s) [noise=true , address=104.36.82.6:4876 , piece_id=XQUCW55LNBVJBY2I73LBSVISZ54JFW4KSSD3SSI4C2QMWEPXSX4Q, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.478579806 (377h3m46.249221141s) [noise=true , address=104.36.82.6:4874 , piece_id=PKM52XEQYB62IF75NYSPLQZNHRMO7QE56QLIHPC7ZIJBRJ3HOVBA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.478878968 (377h3m46.249520303s) [noise=true , address=45.91.116.4:4939 , piece_id=OMGUKZV52SVS2MK4LMVTSJGIEKCNTS7QU36FCXQTN3JIXNWTGN4A, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479123196 (377h3m46.249764531s) [noise=true , address=104.36.82.12:4149 , piece_id=K36PH6SAQRP2EOSPXELGH7KLA3PTLG77JGL5YVCELPFY7SLHLQ5A, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479199021 (377h3m46.249840376s) [noise=true , address=45.91.116.12:4226 , piece_id=VZ7UJ3JWBCXUKX5VAQVOK2KKN6T4BJX2TMHRMN5FUERR5IMN7ZWA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479511829 (377h3m46.250153164s) [noise=true , address=45.91.116.8:4045 , piece_id=MCNAFQZCMOL57CK6WJIBV2HIL65GD3FW7G5PIVKFVBXXUXYFX6LA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479780483 (377h3m46.250421819s) [noise=true , address=104.36.82.8:4970 , piece_id=E6OKLTVW3JMOQOJZUKVC4GBTLSW7CZ2NBVFGO4UUJQ66JBUDE7RA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479798347 (377h3m46.250439672s) [noise=true , address=104.36.82.5:4840 , piece_id=CCMPPQEVRQA2X2LXJ4PSM5ZH2IABEF45G66Q27NV6E4NCSVDOL2A, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480259009 (377h3m46.250900334s) [noise=true , address=104.36.82.16:4294 , piece_id=QVJIAM6GMAJKETPJQPACPSQCR5ITZH5ILHJJWF6LXVEZBNTX56JQ, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480355454 (377h3m46.250996789s) [noise=true , address=77.90.31.11:4553 , piece_id=L7ABAZT6JX265EIDHNPXQXTP3RMXRTPPCMCD76NJTPMBMTCKFIMA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480653393 (377h3m46.251294738s) [noise=true , address=77.90.31.21:4879 , piece_id=LP5SORULTBNCFPGOMLPLDJQ3WTHBXHTJ3MLFXZFW65FW6ITY2BTA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480924812 (377h3m46.251566147s) [noise=true , address=45.91.116.8:4047 , piece_id=JQV6KR6HJDYNN3AASSY2K4B3T673XSXKUYRT6LR3TCB2N43G54ZA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480949319 (377h3m46.251590654s) [noise=true , address=77.90.31.10:4518 , piece_id=ZWOI3W26Y5MBWHP3PIVHYQ67LDLDPQ5TWPPXDJS7LEYW4EVVCORA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.481249524 (377h3m46.251890859s) [noise=true , address=45.91.116.14:4190 , piece_id=MOKLQKDHNXPZGY7FVTH6PQ7QXOEEWMIDBEMLUOPNWJYV4H3WTP6Q, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.481495965 (377h3m46.2521373s) [noise=true , address=77.90.31.21:4878 , piece_id=D7W27EYYNVA7Z5U2XC66PZZLSME3GQZOBFXQKADR2QE7WASPOROQ, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:13.042937288 (377h3m47.813578663s) [upload=301655]: Next returned nil segment
Jul 9 02:14:13.042961194 (377h3m47.813602549s) [upload=301655]: Waiting for error group managing segments...
Jul 9 02:14:13.363736407 (377h3m48.134377742s) [noise=true , address=104.36.82.5:4840 , piece_id=CCMPPQEVRQA2X2LXJ4PSM5ZH2IABEF45G66Q27NV6E4NCSVDOL2A, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.539916948 (377h3m48.310558283s) [noise=true , address=45.91.116.13:4155 , piece_id=D7BUOVLC3Q6T5GX67ZSA6NZNI4RTDVMULDK32D3EAQ2GWGEUL6FQ, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.542285079 (377h3m48.312926414s) [noise=true , address=45.91.116.14:4190 , piece_id=MOKLQKDHNXPZGY7FVTH6PQ7QXOEEWMIDBEMLUOPNWJYV4H3WTP6Q, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.547444790 (377h3m48.318086135s) [noise=true , address=45.91.116.12:4226 , piece_id=VZ7UJ3JWBCXUKX5VAQVOK2KKN6T4BJX2TMHRMN5FUERR5IMN7ZWA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.556717610 (377h3m48.327358945s) [noise=true , address=104.36.82.16:4294 , piece_id=QVJIAM6GMAJKETPJQPACPSQCR5ITZH5ILHJJWF6LXVEZBNTX56JQ, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.565064359 (377h3m48.335705694s) [noise=true , address=77.90.31.21:4879 , piece_id=LP5SORULTBNCFPGOMLPLDJQ3WTHBXHTJ3MLFXZFW65FW6ITY2BTA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.568914063 (377h3m48.339555388s) [noise=true , address=77.90.31.10:4518 , piece_id=ZWOI3W26Y5MBWHP3PIVHYQ67LDLDPQ5TWPPXDJS7LEYW4EVVCORA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.571640700 (377h3m48.342282035s) [noise=true , address=77.90.31.11:4553 , piece_id=L7ABAZT6JX265EIDHNPXQXTP3RMXRTPPCMCD76NJTPMBMTCKFIMA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.576586953 (377h3m48.347228288s) [noise=true , address=77.90.31.21:4878 , piece_id=D7W27EYYNVA7Z5U2XC66PZZLSME3GQZOBFXQKADR2QE7WASPOROQ, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.607484504 (377h3m48.378125839s) [noise=true , address=45.91.116.4:4939 , piece_id=OMGUKZV52SVS2MK4LMVTSJGIEKCNTS7QU36FCXQTN3JIXNWTGN4A, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.618215825 (377h3m48.38885716s) [noise=true , address=45.91.116.8:4047 , piece_id=JQV6KR6HJDYNN3AASSY2K4B3T673XSXKUYRT6LR3TCB2N43G54ZA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.695873867 (377h3m48.466515202s) [noise=true , address=45.91.116.8:4045 , piece_id=MCNAFQZCMOL57CK6WJIBV2HIL65GD3FW7G5PIVKFVBXXUXYFX6LA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:40.587881399 (377h4m15.358522734s) [noise=true , address=104.36.82.8:4970 , piece_id=E6OKLTVW3JMOQOJZUKVC4GBTLSW7CZ2NBVFGO4UUJQ66JBUDE7RA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 188.166.225.170:47376->104.36.82.8:4970: use of closed network connection; write tcp 188.166.225.170:47376->104.36.82.8:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47376->104.36.82.8:4970: use of closed network connection
Jul 9 02:14:40.587919511 (377h4m15.358560846s) [noise=true , address=104.36.82.12:4149 , piece_id=K36PH6SAQRP2EOSPXELGH7KLA3PTLG77JGL5YVCELPFY7SLHLQ5A, seg_pos={0 0}, upload=301655]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:51768->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:51768->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->104.36.82.12:4149: use of closed network connection
Jul 9 02:14:40.587938768 (377h4m15.358580103s) [noise=true , address=104.36.82.6:4874 , piece_id=PKM52XEQYB62IF75NYSPLQZNHRMO7QE56QLIHPC7ZIJBRJ3HOVBA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:56922->104.36.82.6:4874: use of closed network connection; write tcp 188.166.225.170:56922->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56922->104.36.82.6:4874: use of closed network connection
Jul 9 02:14:40.587999454 (377h4m15.358640779s) [noise=true , address=104.36.82.6:4876 , piece_id=XQUCW55LNBVJBY2I73LBSVISZ54JFW4KSSD3SSI4C2QMWEPXSX4Q, seg_pos={0 0}, upload=301655]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:41436->104.36.82.6:4876: use of closed network connection; write tcp 188.166.225.170:41436->104.36.82.6:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41436->104.36.82.6:4876: use of closed network connection
Jul 9 02:14:40.588019132 (377h4m15.358660457s) [seg_pos={0 0}, upload=301655]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:14:40.820632878 (377h4m15.591274213s) [upload=301655]: Done uploading segments...
Jul 9 02:14:40.820657295 (377h4m15.59129862s) [upload=301655]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.797984914s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [.X............X......X..X...X.X.X.X...X...............X.....] duration:
11.590200493s
FAIL (2025-07-09 02:15:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 11.590222425s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [....X......X..........X.............X...............X...X...] duration:
20.233165618s
FAIL (2025-07-09 02:15:33 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
.
Jul 9 02:12:07.627735816 (377h1m42.398377151s) [upload=301626]: Got next segment. Inline: false
Jul 9 02:12:07.627768629 (377h1m42.398409964s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:12:07.627773568 (377h1m42.398414903s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:12:08.075105689 (377h1m42.845747024s) [seg_pos={0 0}, upload=301626]: Begin upload segment...
Jul 9 02:12:08.075172738 (377h1m42.845814073s) [seg_pos={0 0}, upload=301626]: Done begin upload segment.
Jul 9 02:12:08.075200341 (377h1m42.845841676s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075226711 (377h1m42.845868046s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075642747 (377h1m42.846284102s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076099570 (377h1m42.846740905s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076193791 (377h1m42.846835126s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076453428 (377h1m42.847094763s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076686334 (377h1m42.847327669s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076926883 (377h1m42.847568218s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076956250 (377h1m42.847597585s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077181110 (377h1m42.847822445s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077198733 (377h1m42.847840068s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077433213 (377h1m42.848074548s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077452068 (377h1m42.848093403s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077655618 (377h1m42.848296953s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077884866 (377h1m42.848526201s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078100028 (377h1m42.848741363s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078310551 (377h1m42.848951886s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078506095 (377h1m42.84914743s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:09.631539904 (377h1m44.402181239s) [upload=301626]: Next returned nil segment
Jul 9 02:12:09.631556275 (377h1m44.4021976s) [upload=301626]: Waiting for error group managing segments...
Jul 9 02:12:10.026556908 (377h1m44.797198243s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.029581435 (377h1m44.80022277s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.030348031 (377h1m44.800989366s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.037300182 (377h1m44.807941517s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.038590761 (377h1m44.809232096s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.040614022 (377h1m44.811255347s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.243165042 (377h1m45.013806377s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.294115297 (377h1m45.064756632s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:12.752127764 (377h1m47.522769099s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:29.819338082 (377h2m4.589979417s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:35.349456003 (377h2m10.120097338s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:40.626537094 (377h2m15.397178419s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:42.766847833 (377h2m17.537489168s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:13:06.981429964 (377h2m41.752071299s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:15mgZMkU28J1JwrRd4XbvZFyjp8G7CTBkJCxbzUYqKeSeqCZ9F, address:104.36.82.7:4944): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981472496 (377h2m41.752113821s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; EOF
Jul 9 02:13:06.981498496 (377h2m41.752139821s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:12pHU3Xp1sDgnmYMcbWZD2DivxdEUzqnBVJaiRaQipuRy2d75u5, address:104.36.82.18:4375): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981567808 (377h2m41.752209143s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981595070 (377h2m41.752236395s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection
Jul 9 02:13:06.981611983 (377h2m41.752253318s) [seg_pos={0 0}, upload=301626]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:13:07.214001409 (377h2m41.984642744s) [upload=301626]: Done uploading segments...
Jul 9 02:13:07.214031306 (377h2m41.984672641s) [upload=301626]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 20.23318276s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [...................X.................X......X.........XXX.X.] duration:
5.891580215s
FAIL (2025-07-09 02:15:18 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
tchRequestItem_ObjectCommit
Jul 9 02:13:15.533673746 (377h2m50.304315071s) [upload=301640]: Done uploading segments...
Jul 9 02:13:15.533714053 (377h2m50.304355388s) [upload=301640]: Upload finished. err: <nil>
Jul 9 02:14:11.239664264 (377h3m46.010305599s) [upload=301655]: Starting upload
Jul 9 02:14:11.239720071 (377h3m46.010361406s) [upload=301655]: Done starting upload
Jul 9 02:14:11.239901257 (377h3m46.010542592s) [upload=301655]: Uploading segments...
Jul 9 02:14:11.239936835 (377h3m46.01057816s) [upload=301655]: Got next segment. Inline: false
Jul 9 02:14:11.239961161 (377h3m46.010602496s) [upload=301655]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:14:11.239965630 (377h3m46.010606965s) [upload=301655]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:14:11.478079609 (377h3m46.248720944s) [seg_pos={0 0}, upload=301655]: Begin upload segment...
Jul 9 02:14:11.478117833 (377h3m46.248759158s) [seg_pos={0 0}, upload=301655]: Done begin upload segment.
Jul 9 02:14:11.478140505 (377h3m46.24878184s) [noise=true , address=45.91.116.13:4155 , piece_id=D7BUOVLC3Q6T5GX67ZSA6NZNI4RTDVMULDK32D3EAQ2GWGEUL6FQ, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.478246749 (377h3m46.248888084s) [noise=true , address=104.36.82.6:4876 , piece_id=XQUCW55LNBVJBY2I73LBSVISZ54JFW4KSSD3SSI4C2QMWEPXSX4Q, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.478579806 (377h3m46.249221141s) [noise=true , address=104.36.82.6:4874 , piece_id=PKM52XEQYB62IF75NYSPLQZNHRMO7QE56QLIHPC7ZIJBRJ3HOVBA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.478878968 (377h3m46.249520303s) [noise=true , address=45.91.116.4:4939 , piece_id=OMGUKZV52SVS2MK4LMVTSJGIEKCNTS7QU36FCXQTN3JIXNWTGN4A, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479123196 (377h3m46.249764531s) [noise=true , address=104.36.82.12:4149 , piece_id=K36PH6SAQRP2EOSPXELGH7KLA3PTLG77JGL5YVCELPFY7SLHLQ5A, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479199021 (377h3m46.249840376s) [noise=true , address=45.91.116.12:4226 , piece_id=VZ7UJ3JWBCXUKX5VAQVOK2KKN6T4BJX2TMHRMN5FUERR5IMN7ZWA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479511829 (377h3m46.250153164s) [noise=true , address=45.91.116.8:4045 , piece_id=MCNAFQZCMOL57CK6WJIBV2HIL65GD3FW7G5PIVKFVBXXUXYFX6LA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479780483 (377h3m46.250421819s) [noise=true , address=104.36.82.8:4970 , piece_id=E6OKLTVW3JMOQOJZUKVC4GBTLSW7CZ2NBVFGO4UUJQ66JBUDE7RA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479798347 (377h3m46.250439672s) [noise=true , address=104.36.82.5:4840 , piece_id=CCMPPQEVRQA2X2LXJ4PSM5ZH2IABEF45G66Q27NV6E4NCSVDOL2A, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480259009 (377h3m46.250900334s) [noise=true , address=104.36.82.16:4294 , piece_id=QVJIAM6GMAJKETPJQPACPSQCR5ITZH5ILHJJWF6LXVEZBNTX56JQ, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480355454 (377h3m46.250996789s) [noise=true , address=77.90.31.11:4553 , piece_id=L7ABAZT6JX265EIDHNPXQXTP3RMXRTPPCMCD76NJTPMBMTCKFIMA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480653393 (377h3m46.251294738s) [noise=true , address=77.90.31.21:4879 , piece_id=LP5SORULTBNCFPGOMLPLDJQ3WTHBXHTJ3MLFXZFW65FW6ITY2BTA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480924812 (377h3m46.251566147s) [noise=true , address=45.91.116.8:4047 , piece_id=JQV6KR6HJDYNN3AASSY2K4B3T673XSXKUYRT6LR3TCB2N43G54ZA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480949319 (377h3m46.251590654s) [noise=true , address=77.90.31.10:4518 , piece_id=ZWOI3W26Y5MBWHP3PIVHYQ67LDLDPQ5TWPPXDJS7LEYW4EVVCORA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.481249524 (377h3m46.251890859s) [noise=true , address=45.91.116.14:4190 , piece_id=MOKLQKDHNXPZGY7FVTH6PQ7QXOEEWMIDBEMLUOPNWJYV4H3WTP6Q, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.481495965 (377h3m46.2521373s) [noise=true , address=77.90.31.21:4878 , piece_id=D7W27EYYNVA7Z5U2XC66PZZLSME3GQZOBFXQKADR2QE7WASPOROQ, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:13.042937288 (377h3m47.813578663s) [upload=301655]: Next returned nil segment
Jul 9 02:14:13.042961194 (377h3m47.813602549s) [upload=301655]: Waiting for error group managing segments...
Jul 9 02:14:13.363736407 (377h3m48.134377742s) [noise=true , address=104.36.82.5:4840 , piece_id=CCMPPQEVRQA2X2LXJ4PSM5ZH2IABEF45G66Q27NV6E4NCSVDOL2A, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.539916948 (377h3m48.310558283s) [noise=true , address=45.91.116.13:4155 , piece_id=D7BUOVLC3Q6T5GX67ZSA6NZNI4RTDVMULDK32D3EAQ2GWGEUL6FQ, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.542285079 (377h3m48.312926414s) [noise=true , address=45.91.116.14:4190 , piece_id=MOKLQKDHNXPZGY7FVTH6PQ7QXOEEWMIDBEMLUOPNWJYV4H3WTP6Q, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.547444790 (377h3m48.318086135s) [noise=true , address=45.91.116.12:4226 , piece_id=VZ7UJ3JWBCXUKX5VAQVOK2KKN6T4BJX2TMHRMN5FUERR5IMN7ZWA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.556717610 (377h3m48.327358945s) [noise=true , address=104.36.82.16:4294 , piece_id=QVJIAM6GMAJKETPJQPACPSQCR5ITZH5ILHJJWF6LXVEZBNTX56JQ, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.565064359 (377h3m48.335705694s) [noise=true , address=77.90.31.21:4879 , piece_id=LP5SORULTBNCFPGOMLPLDJQ3WTHBXHTJ3MLFXZFW65FW6ITY2BTA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.568914063 (377h3m48.339555388s) [noise=true , address=77.90.31.10:4518 , piece_id=ZWOI3W26Y5MBWHP3PIVHYQ67LDLDPQ5TWPPXDJS7LEYW4EVVCORA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.571640700 (377h3m48.342282035s) [noise=true , address=77.90.31.11:4553 , piece_id=L7ABAZT6JX265EIDHNPXQXTP3RMXRTPPCMCD76NJTPMBMTCKFIMA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.576586953 (377h3m48.347228288s) [noise=true , address=77.90.31.21:4878 , piece_id=D7W27EYYNVA7Z5U2XC66PZZLSME3GQZOBFXQKADR2QE7WASPOROQ, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.607484504 (377h3m48.378125839s) [noise=true , address=45.91.116.4:4939 , piece_id=OMGUKZV52SVS2MK4LMVTSJGIEKCNTS7QU36FCXQTN3JIXNWTGN4A, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.618215825 (377h3m48.38885716s) [noise=true , address=45.91.116.8:4047 , piece_id=JQV6KR6HJDYNN3AASSY2K4B3T673XSXKUYRT6LR3TCB2N43G54ZA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.695873867 (377h3m48.466515202s) [noise=true , address=45.91.116.8:4045 , piece_id=MCNAFQZCMOL57CK6WJIBV2HIL65GD3FW7G5PIVKFVBXXUXYFX6LA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:40.587881399 (377h4m15.358522734s) [noise=true , address=104.36.82.8:4970 , piece_id=E6OKLTVW3JMOQOJZUKVC4GBTLSW7CZ2NBVFGO4UUJQ66JBUDE7RA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 188.166.225.170:47376->104.36.82.8:4970: use of closed network connection; write tcp 188.166.225.170:47376->104.36.82.8:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47376->104.36.82.8:4970: use of closed network connection
Jul 9 02:14:40.587919511 (377h4m15.358560846s) [noise=true , address=104.36.82.12:4149 , piece_id=K36PH6SAQRP2EOSPXELGH7KLA3PTLG77JGL5YVCELPFY7SLHLQ5A, seg_pos={0 0}, upload=301655]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:51768->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:51768->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->104.36.82.12:4149: use of closed network connection
Jul 9 02:14:40.587938768 (377h4m15.358580103s) [noise=true , address=104.36.82.6:4874 , piece_id=PKM52XEQYB62IF75NYSPLQZNHRMO7QE56QLIHPC7ZIJBRJ3HOVBA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:56922->104.36.82.6:4874: use of closed network connection; write tcp 188.166.225.170:56922->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56922->104.36.82.6:4874: use of closed network connection
Jul 9 02:14:40.587999454 (377h4m15.358640779s) [noise=true , address=104.36.82.6:4876 , piece_id=XQUCW55LNBVJBY2I73LBSVISZ54JFW4KSSD3SSI4C2QMWEPXSX4Q, seg_pos={0 0}, upload=301655]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:41436->104.36.82.6:4876: use of closed network connection; write tcp 188.166.225.170:41436->104.36.82.6:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41436->104.36.82.6:4876: use of closed network connection
Jul 9 02:14:40.588019132 (377h4m15.358660457s) [seg_pos={0 0}, upload=301655]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:14:40.820632878 (377h4m15.591274213s) [upload=301655]: Done uploading segments...
Jul 9 02:14:40.820657295 (377h4m15.59129862s) [upload=301655]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.891616804s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.....X......X..........X.............X...............X...X..] duration:
59.512469295s
FAIL (2025-07-09 02:15:12 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
.
Jul 9 02:12:07.627735816 (377h1m42.398377151s) [upload=301626]: Got next segment. Inline: false
Jul 9 02:12:07.627768629 (377h1m42.398409964s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:12:07.627773568 (377h1m42.398414903s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:12:08.075105689 (377h1m42.845747024s) [seg_pos={0 0}, upload=301626]: Begin upload segment...
Jul 9 02:12:08.075172738 (377h1m42.845814073s) [seg_pos={0 0}, upload=301626]: Done begin upload segment.
Jul 9 02:12:08.075200341 (377h1m42.845841676s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075226711 (377h1m42.845868046s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075642747 (377h1m42.846284102s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076099570 (377h1m42.846740905s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076193791 (377h1m42.846835126s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076453428 (377h1m42.847094763s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076686334 (377h1m42.847327669s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076926883 (377h1m42.847568218s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076956250 (377h1m42.847597585s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077181110 (377h1m42.847822445s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077198733 (377h1m42.847840068s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077433213 (377h1m42.848074548s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077452068 (377h1m42.848093403s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077655618 (377h1m42.848296953s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077884866 (377h1m42.848526201s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078100028 (377h1m42.848741363s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078310551 (377h1m42.848951886s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078506095 (377h1m42.84914743s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:09.631539904 (377h1m44.402181239s) [upload=301626]: Next returned nil segment
Jul 9 02:12:09.631556275 (377h1m44.4021976s) [upload=301626]: Waiting for error group managing segments...
Jul 9 02:12:10.026556908 (377h1m44.797198243s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.029581435 (377h1m44.80022277s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.030348031 (377h1m44.800989366s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.037300182 (377h1m44.807941517s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.038590761 (377h1m44.809232096s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.040614022 (377h1m44.811255347s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.243165042 (377h1m45.013806377s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.294115297 (377h1m45.064756632s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:12.752127764 (377h1m47.522769099s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:29.819338082 (377h2m4.589979417s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:35.349456003 (377h2m10.120097338s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:40.626537094 (377h2m15.397178419s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:42.766847833 (377h2m17.537489168s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:13:06.981429964 (377h2m41.752071299s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:15mgZMkU28J1JwrRd4XbvZFyjp8G7CTBkJCxbzUYqKeSeqCZ9F, address:104.36.82.7:4944): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981472496 (377h2m41.752113821s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; EOF
Jul 9 02:13:06.981498496 (377h2m41.752139821s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:12pHU3Xp1sDgnmYMcbWZD2DivxdEUzqnBVJaiRaQipuRy2d75u5, address:104.36.82.18:4375): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981567808 (377h2m41.752209143s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981595070 (377h2m41.752236395s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection
Jul 9 02:13:06.981611983 (377h2m41.752253318s) [seg_pos={0 0}, upload=301626]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:13:07.214001409 (377h2m41.984642744s) [upload=301626]: Done uploading segments...
Jul 9 02:13:07.214031306 (377h2m41.984672641s) [upload=301626]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 59.51249853s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [..X............X......X..X...X.X.X.X...X...............X....] duration:
12.380479399s
FAIL (2025-07-09 02:15:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.380494027s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [....................X.................X......X.........XXX.X] duration:
30.233950221s
FAIL (2025-07-09 02:14:40 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
tchRequestItem_ObjectCommit
Jul 9 02:13:15.533673746 (377h2m50.304315071s) [upload=301640]: Done uploading segments...
Jul 9 02:13:15.533714053 (377h2m50.304355388s) [upload=301640]: Upload finished. err: <nil>
Jul 9 02:14:11.239664264 (377h3m46.010305599s) [upload=301655]: Starting upload
Jul 9 02:14:11.239720071 (377h3m46.010361406s) [upload=301655]: Done starting upload
Jul 9 02:14:11.239901257 (377h3m46.010542592s) [upload=301655]: Uploading segments...
Jul 9 02:14:11.239936835 (377h3m46.01057816s) [upload=301655]: Got next segment. Inline: false
Jul 9 02:14:11.239961161 (377h3m46.010602496s) [upload=301655]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:14:11.239965630 (377h3m46.010606965s) [upload=301655]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:14:11.478079609 (377h3m46.248720944s) [seg_pos={0 0}, upload=301655]: Begin upload segment...
Jul 9 02:14:11.478117833 (377h3m46.248759158s) [seg_pos={0 0}, upload=301655]: Done begin upload segment.
Jul 9 02:14:11.478140505 (377h3m46.24878184s) [noise=true , address=45.91.116.13:4155 , piece_id=D7BUOVLC3Q6T5GX67ZSA6NZNI4RTDVMULDK32D3EAQ2GWGEUL6FQ, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.478246749 (377h3m46.248888084s) [noise=true , address=104.36.82.6:4876 , piece_id=XQUCW55LNBVJBY2I73LBSVISZ54JFW4KSSD3SSI4C2QMWEPXSX4Q, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.478579806 (377h3m46.249221141s) [noise=true , address=104.36.82.6:4874 , piece_id=PKM52XEQYB62IF75NYSPLQZNHRMO7QE56QLIHPC7ZIJBRJ3HOVBA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.478878968 (377h3m46.249520303s) [noise=true , address=45.91.116.4:4939 , piece_id=OMGUKZV52SVS2MK4LMVTSJGIEKCNTS7QU36FCXQTN3JIXNWTGN4A, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479123196 (377h3m46.249764531s) [noise=true , address=104.36.82.12:4149 , piece_id=K36PH6SAQRP2EOSPXELGH7KLA3PTLG77JGL5YVCELPFY7SLHLQ5A, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479199021 (377h3m46.249840376s) [noise=true , address=45.91.116.12:4226 , piece_id=VZ7UJ3JWBCXUKX5VAQVOK2KKN6T4BJX2TMHRMN5FUERR5IMN7ZWA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479511829 (377h3m46.250153164s) [noise=true , address=45.91.116.8:4045 , piece_id=MCNAFQZCMOL57CK6WJIBV2HIL65GD3FW7G5PIVKFVBXXUXYFX6LA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479780483 (377h3m46.250421819s) [noise=true , address=104.36.82.8:4970 , piece_id=E6OKLTVW3JMOQOJZUKVC4GBTLSW7CZ2NBVFGO4UUJQ66JBUDE7RA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.479798347 (377h3m46.250439672s) [noise=true , address=104.36.82.5:4840 , piece_id=CCMPPQEVRQA2X2LXJ4PSM5ZH2IABEF45G66Q27NV6E4NCSVDOL2A, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480259009 (377h3m46.250900334s) [noise=true , address=104.36.82.16:4294 , piece_id=QVJIAM6GMAJKETPJQPACPSQCR5ITZH5ILHJJWF6LXVEZBNTX56JQ, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480355454 (377h3m46.250996789s) [noise=true , address=77.90.31.11:4553 , piece_id=L7ABAZT6JX265EIDHNPXQXTP3RMXRTPPCMCD76NJTPMBMTCKFIMA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480653393 (377h3m46.251294738s) [noise=true , address=77.90.31.21:4879 , piece_id=LP5SORULTBNCFPGOMLPLDJQ3WTHBXHTJ3MLFXZFW65FW6ITY2BTA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480924812 (377h3m46.251566147s) [noise=true , address=45.91.116.8:4047 , piece_id=JQV6KR6HJDYNN3AASSY2K4B3T673XSXKUYRT6LR3TCB2N43G54ZA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.480949319 (377h3m46.251590654s) [noise=true , address=77.90.31.10:4518 , piece_id=ZWOI3W26Y5MBWHP3PIVHYQ67LDLDPQ5TWPPXDJS7LEYW4EVVCORA, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.481249524 (377h3m46.251890859s) [noise=true , address=45.91.116.14:4190 , piece_id=MOKLQKDHNXPZGY7FVTH6PQ7QXOEEWMIDBEMLUOPNWJYV4H3WTP6Q, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:11.481495965 (377h3m46.2521373s) [noise=true , address=77.90.31.21:4878 , piece_id=D7W27EYYNVA7Z5U2XC66PZZLSME3GQZOBFXQKADR2QE7WASPOROQ, seg_pos={0 0}, upload=301655]: Uploading piece...
Jul 9 02:14:13.042937288 (377h3m47.813578663s) [upload=301655]: Next returned nil segment
Jul 9 02:14:13.042961194 (377h3m47.813602549s) [upload=301655]: Waiting for error group managing segments...
Jul 9 02:14:13.363736407 (377h3m48.134377742s) [noise=true , address=104.36.82.5:4840 , piece_id=CCMPPQEVRQA2X2LXJ4PSM5ZH2IABEF45G66Q27NV6E4NCSVDOL2A, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.539916948 (377h3m48.310558283s) [noise=true , address=45.91.116.13:4155 , piece_id=D7BUOVLC3Q6T5GX67ZSA6NZNI4RTDVMULDK32D3EAQ2GWGEUL6FQ, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.542285079 (377h3m48.312926414s) [noise=true , address=45.91.116.14:4190 , piece_id=MOKLQKDHNXPZGY7FVTH6PQ7QXOEEWMIDBEMLUOPNWJYV4H3WTP6Q, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.547444790 (377h3m48.318086135s) [noise=true , address=45.91.116.12:4226 , piece_id=VZ7UJ3JWBCXUKX5VAQVOK2KKN6T4BJX2TMHRMN5FUERR5IMN7ZWA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.556717610 (377h3m48.327358945s) [noise=true , address=104.36.82.16:4294 , piece_id=QVJIAM6GMAJKETPJQPACPSQCR5ITZH5ILHJJWF6LXVEZBNTX56JQ, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.565064359 (377h3m48.335705694s) [noise=true , address=77.90.31.21:4879 , piece_id=LP5SORULTBNCFPGOMLPLDJQ3WTHBXHTJ3MLFXZFW65FW6ITY2BTA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.568914063 (377h3m48.339555388s) [noise=true , address=77.90.31.10:4518 , piece_id=ZWOI3W26Y5MBWHP3PIVHYQ67LDLDPQ5TWPPXDJS7LEYW4EVVCORA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.571640700 (377h3m48.342282035s) [noise=true , address=77.90.31.11:4553 , piece_id=L7ABAZT6JX265EIDHNPXQXTP3RMXRTPPCMCD76NJTPMBMTCKFIMA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.576586953 (377h3m48.347228288s) [noise=true , address=77.90.31.21:4878 , piece_id=D7W27EYYNVA7Z5U2XC66PZZLSME3GQZOBFXQKADR2QE7WASPOROQ, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.607484504 (377h3m48.378125839s) [noise=true , address=45.91.116.4:4939 , piece_id=OMGUKZV52SVS2MK4LMVTSJGIEKCNTS7QU36FCXQTN3JIXNWTGN4A, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.618215825 (377h3m48.38885716s) [noise=true , address=45.91.116.8:4047 , piece_id=JQV6KR6HJDYNN3AASSY2K4B3T673XSXKUYRT6LR3TCB2N43G54ZA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:13.695873867 (377h3m48.466515202s) [noise=true , address=45.91.116.8:4045 , piece_id=MCNAFQZCMOL57CK6WJIBV2HIL65GD3FW7G5PIVKFVBXXUXYFX6LA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: <nil>
Jul 9 02:14:40.587881399 (377h4m15.358522734s) [noise=true , address=104.36.82.8:4970 , piece_id=E6OKLTVW3JMOQOJZUKVC4GBTLSW7CZ2NBVFGO4UUJQ66JBUDE7RA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 188.166.225.170:47376->104.36.82.8:4970: use of closed network connection; write tcp 188.166.225.170:47376->104.36.82.8:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47376->104.36.82.8:4970: use of closed network connection
Jul 9 02:14:40.587919511 (377h4m15.358560846s) [noise=true , address=104.36.82.12:4149 , piece_id=K36PH6SAQRP2EOSPXELGH7KLA3PTLG77JGL5YVCELPFY7SLHLQ5A, seg_pos={0 0}, upload=301655]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:51768->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:51768->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->104.36.82.12:4149: use of closed network connection
Jul 9 02:14:40.587938768 (377h4m15.358580103s) [noise=true , address=104.36.82.6:4874 , piece_id=PKM52XEQYB62IF75NYSPLQZNHRMO7QE56QLIHPC7ZIJBRJ3HOVBA, seg_pos={0 0}, upload=301655]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:56922->104.36.82.6:4874: use of closed network connection; write tcp 188.166.225.170:56922->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56922->104.36.82.6:4874: use of closed network connection
Jul 9 02:14:40.587999454 (377h4m15.358640779s) [noise=true , address=104.36.82.6:4876 , piece_id=XQUCW55LNBVJBY2I73LBSVISZ54JFW4KSSD3SSI4C2QMWEPXSX4Q, seg_pos={0 0}, upload=301655]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:41436->104.36.82.6:4876: use of closed network connection; write tcp 188.166.225.170:41436->104.36.82.6:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41436->104.36.82.6:4876: use of closed network connection
Jul 9 02:14:40.588019132 (377h4m15.358660457s) [seg_pos={0 0}, upload=301655]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:14:40.820632878 (377h4m15.591274213s) [upload=301655]: Done uploading segments...
Jul 9 02:14:40.820657295 (377h4m15.59129862s) [upload=301655]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.233974858s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
19 out of 60 is failed (threshold=3) [.XX.XX..X......XXX....XX.....XX...X.XX........X.X......X..X.] duration:
37.886101017s
FAIL (2025-07-09 02:14:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 37.886112159s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [X..X............X......X..X...X.X.X.X...X...............X...] duration:
10.08697405s
FAIL (2025-07-09 02:14:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.086992316s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
19 out of 60 is failed (threshold=3) [..XX.XX..X......XXX....XX.....XX...X.XX........X.X......X..X] duration:
1m0.000591299s
FAIL (2025-07-09 02:14:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000603201s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [......X......X..........X.............X...............X...X.] duration:
48.90285607s
FAIL (2025-07-09 02:13:56 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
.
Jul 9 02:12:07.627735816 (377h1m42.398377151s) [upload=301626]: Got next segment. Inline: false
Jul 9 02:12:07.627768629 (377h1m42.398409964s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:12:07.627773568 (377h1m42.398414903s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:12:08.075105689 (377h1m42.845747024s) [seg_pos={0 0}, upload=301626]: Begin upload segment...
Jul 9 02:12:08.075172738 (377h1m42.845814073s) [seg_pos={0 0}, upload=301626]: Done begin upload segment.
Jul 9 02:12:08.075200341 (377h1m42.845841676s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075226711 (377h1m42.845868046s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075642747 (377h1m42.846284102s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076099570 (377h1m42.846740905s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076193791 (377h1m42.846835126s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076453428 (377h1m42.847094763s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076686334 (377h1m42.847327669s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076926883 (377h1m42.847568218s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076956250 (377h1m42.847597585s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077181110 (377h1m42.847822445s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077198733 (377h1m42.847840068s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077433213 (377h1m42.848074548s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077452068 (377h1m42.848093403s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077655618 (377h1m42.848296953s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077884866 (377h1m42.848526201s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078100028 (377h1m42.848741363s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078310551 (377h1m42.848951886s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078506095 (377h1m42.84914743s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:09.631539904 (377h1m44.402181239s) [upload=301626]: Next returned nil segment
Jul 9 02:12:09.631556275 (377h1m44.4021976s) [upload=301626]: Waiting for error group managing segments...
Jul 9 02:12:10.026556908 (377h1m44.797198243s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.029581435 (377h1m44.80022277s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.030348031 (377h1m44.800989366s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.037300182 (377h1m44.807941517s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.038590761 (377h1m44.809232096s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.040614022 (377h1m44.811255347s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.243165042 (377h1m45.013806377s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.294115297 (377h1m45.064756632s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:12.752127764 (377h1m47.522769099s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:29.819338082 (377h2m4.589979417s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:35.349456003 (377h2m10.120097338s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:40.626537094 (377h2m15.397178419s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:42.766847833 (377h2m17.537489168s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:13:06.981429964 (377h2m41.752071299s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:15mgZMkU28J1JwrRd4XbvZFyjp8G7CTBkJCxbzUYqKeSeqCZ9F, address:104.36.82.7:4944): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981472496 (377h2m41.752113821s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; EOF
Jul 9 02:13:06.981498496 (377h2m41.752139821s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:12pHU3Xp1sDgnmYMcbWZD2DivxdEUzqnBVJaiRaQipuRy2d75u5, address:104.36.82.18:4375): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981567808 (377h2m41.752209143s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981595070 (377h2m41.752236395s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection
Jul 9 02:13:06.981611983 (377h2m41.752253318s) [seg_pos={0 0}, upload=301626]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:13:07.214001409 (377h2m41.984642744s) [upload=301626]: Done uploading segments...
Jul 9 02:13:07.214031306 (377h2m41.984672641s) [upload=301626]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 48.902869836s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.X..X............X......X..X...X.X.X.X...X...............X..] duration:
23.175869391s
FAIL (2025-07-09 02:13:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 23.175891244s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.....................X.................X......X.........XXX.] duration:
5.871961514s
FAIL (2025-07-09 02:13:18 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
71 (377h1m36.523589407s) [upload=301625]: Uploading segments...
Jul 9 02:12:01.753019818 (377h1m36.523661153s) [upload=301625]: Got next segment. Inline: false
Jul 9 02:12:01.753045758 (377h1m36.523687093s) [upload=301625]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:12:01.753050808 (377h1m36.523692143s) [upload=301625]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:12:01.989838026 (377h1m36.760479361s) [seg_pos={0 0}, upload=301625]: Begin upload segment...
Jul 9 02:12:01.989881920 (377h1m36.760523255s) [seg_pos={0 0}, upload=301625]: Done begin upload segment.
Jul 9 02:12:01.989897190 (377h1m36.760538525s) [noise=true , address=45.91.116.14:4191 , piece_id=C5GROPMRE7AYIG4SDMVDH63VJUKCYMW6U7EKM3MVIOUZBC6DUQMQ, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.989905004 (377h1m36.760546349s) [noise=true , address=45.91.116.7:4011 , piece_id=7H4X3T745B5KFEQ4LJGOLBFDKRJY7WAMKA2P3UT3GUHPJQ7EVPGQ, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.990218293 (377h1m36.760859629s) [noise=true , address=77.90.31.16:4734 , piece_id=K4LSJTW4FHH3HURFBIB7OONWE32CZGTO636ESXIWTNVCDOXEM3CA, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.990226188 (377h1m36.760867523s) [noise=true , address=45.91.116.5:4297 , piece_id=AJZOVXX5X25H75ESLPFCPXA57JLKZJWJS67MIHLP6XWNNABXEURA, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.990459024 (377h1m36.761100359s) [noise=true , address=45.91.116.6:4974 , piece_id=CWZNS2UQYFSKCAAY2IA55TAU5DQV6P7I5TJSA4IULP5V4KJJ6TBQ, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.990525110 (377h1m36.761166445s) [noise=true , address=104.36.82.13:4186 , piece_id=7GMI3HN2WNK6CRLORWPCUMFSF5URMWE7OZV5AGBQBUTCTJ5TMD7Q, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.990686329 (377h1m36.761327664s) [noise=true , address=45.91.116.11:4119 , piece_id=D7SQQI3L6XV7QDSUGUXU6B2KA5SQW5VN7ZU2LWVKZSBI2ENQ46YQ, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.990920928 (377h1m36.761562263s) [noise=true , address=45.91.116.8:4045 , piece_id=RJ6ECPUGUOXRJCCAZWZ2UZSDA4XC7QWTPLVA2M435VGP6ZFVQR4A, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.991140709 (377h1m36.761782044s) [noise=true , address=104.36.82.2:4730 , piece_id=DLZH3HN2DJ4RG7IBONW5IRKJFGQBSYZCBEX6H32KR5TTFPIKHEXQ, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.991363214 (377h1m36.762004549s) [noise=true , address=77.90.31.17:4772 , piece_id=3AP77S5YRSLMEEWS4KIYYJ6BXYVIEW5ESRFXVZIS4IL5LT77TV2Q, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.991580861 (377h1m36.762222196s) [noise=true , address=45.91.116.10:4082 , piece_id=R7RFWF454POA57PH4QJDNOR4CDRSVWKE4GLNSAUQIRH6YBBQBOLQ, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.991778940 (377h1m36.762420275s) [noise=true , address=45.91.116.13:4155 , piece_id=MP7WL4IKLOQK53RSTK2SKYKIAUK5DDKVBEWRLFKJBAQGTWQMFM7A, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.992034609 (377h1m36.762675944s) [noise=true , address=104.36.82.12:4150 , piece_id=5AFHJ72DS2GJYUNACDBG4SRZPYHUGOUSEAJNWKH2362UE2S6KZ5A, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.992236064 (377h1m36.762877399s) [noise=true , address=45.91.116.13:4153 , piece_id=GZX6EVLDMA4LI436AIHADCESLSYPNBXW6S4QBOXJMDSVSFM4KKHA, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.993996933 (377h1m36.764638268s) [noise=true , address=104.36.82.2:4731 , piece_id=CIWRKY5Q6PSNPQZVOXBRJ2WCDSKV2VB2YXYPMN4CVKA7POC3CXKQ, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.994263944 (377h1m36.764905279s) [noise=true , address=104.36.82.14:4222 , piece_id=3NTL5ZFBRX2JSPOGHDURULC3X7IMWSMJK52MW7MJ4QF3YB7OJW4A, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:03.750924051 (377h1m38.521565386s) [upload=301625]: Next returned nil segment
Jul 9 02:12:03.750940513 (377h1m38.521581848s) [upload=301625]: Waiting for error group managing segments...
Jul 9 02:12:04.047569006 (377h1m38.818210341s) [noise=true , address=45.91.116.10:4082 , piece_id=R7RFWF454POA57PH4QJDNOR4CDRSVWKE4GLNSAUQIRH6YBBQBOLQ, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:04.049842796 (377h1m38.820484131s) [noise=true , address=45.91.116.13:4153 , piece_id=GZX6EVLDMA4LI436AIHADCESLSYPNBXW6S4QBOXJMDSVSFM4KKHA, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:04.050277458 (377h1m38.820918793s) [noise=true , address=45.91.116.14:4191 , piece_id=C5GROPMRE7AYIG4SDMVDH63VJUKCYMW6U7EKM3MVIOUZBC6DUQMQ, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:04.050419810 (377h1m38.821061145s) [noise=true , address=45.91.116.7:4011 , piece_id=7H4X3T745B5KFEQ4LJGOLBFDKRJY7WAMKA2P3UT3GUHPJQ7EVPGQ, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:04.054000681 (377h1m38.824642006s) [noise=true , address=45.91.116.5:4297 , piece_id=AJZOVXX5X25H75ESLPFCPXA57JLKZJWJS67MIHLP6XWNNABXEURA, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:04.058369419 (377h1m38.829010754s) [noise=true , address=45.91.116.13:4155 , piece_id=MP7WL4IKLOQK53RSTK2SKYKIAUK5DDKVBEWRLFKJBAQGTWQMFM7A, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:04.063800109 (377h1m38.834441444s) [noise=true , address=45.91.116.11:4119 , piece_id=D7SQQI3L6XV7QDSUGUXU6B2KA5SQW5VN7ZU2LWVKZSBI2ENQ46YQ, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:04.079959624 (377h1m38.850600959s) [noise=true , address=77.90.31.17:4772 , piece_id=3AP77S5YRSLMEEWS4KIYYJ6BXYVIEW5ESRFXVZIS4IL5LT77TV2Q, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:04.094478832 (377h1m38.865120167s) [noise=true , address=77.90.31.16:4734 , piece_id=K4LSJTW4FHH3HURFBIB7OONWE32CZGTO636ESXIWTNVCDOXEM3CA, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:04.245456283 (377h1m39.016097618s) [noise=true , address=45.91.116.6:4974 , piece_id=CWZNS2UQYFSKCAAY2IA55TAU5DQV6P7I5TJSA4IULP5V4KJJ6TBQ, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:05.376301634 (377h1m40.146942969s) [noise=true , address=45.91.116.8:4045 , piece_id=RJ6ECPUGUOXRJCCAZWZ2UZSDA4XC7QWTPLVA2M435VGP6ZFVQR4A, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:31.108895367 (377h2m5.879536702s) [noise=true , address=104.36.82.12:4150 , piece_id=5AFHJ72DS2GJYUNACDBG4SRZPYHUGOUSEAJNWKH2362UE2S6KZ5A, seg_pos={0 0}, upload=301625]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:43048->104.36.82.12:4150: use of closed network connection; write tcp 188.166.225.170:43048->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43048->104.36.82.12:4150: use of closed network connection
Jul 9 02:12:31.108946284 (377h2m5.879587619s) [noise=true , address=104.36.82.2:4730 , piece_id=DLZH3HN2DJ4RG7IBONW5IRKJFGQBSYZCBEX6H32KR5TTFPIKHEXQ, seg_pos={0 0}, upload=301625]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:43754->104.36.82.2:4730: use of closed network connection; write tcp 188.166.225.170:43754->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43754->104.36.82.2:4730: use of closed network connection
Jul 9 02:12:31.108983765 (377h2m5.879625101s) [noise=true , address=104.36.82.14:4222 , piece_id=3NTL5ZFBRX2JSPOGHDURULC3X7IMWSMJK52MW7MJ4QF3YB7OJW4A, seg_pos={0 0}, upload=301625]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:37012->104.36.82.14:4222: use of closed network connection; write tcp 188.166.225.170:37012->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37012->104.36.82.14:4222: use of closed network connection
Jul 9 02:12:31.109017410 (377h2m5.879658745s) [noise=true , address=104.36.82.2:4731 , piece_id=CIWRKY5Q6PSNPQZVOXBRJ2WCDSKV2VB2YXYPMN4CVKA7POC3CXKQ, seg_pos={0 0}, upload=301625]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:54750->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:54750->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54750->104.36.82.2:4731: use of closed network connection
Jul 9 02:12:31.109051516 (377h2m5.879692841s) [noise=true , address=104.36.82.13:4186 , piece_id=7GMI3HN2WNK6CRLORWPCUMFSF5URMWE7OZV5AGBQBUTCTJ5TMD7Q, seg_pos={0 0}, upload=301625]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 188.166.225.170:33706->104.36.82.13:4186: use of closed network connection; write tcp 188.166.225.170:33706->104.36.82.13:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33706->104.36.82.13:4186: use of closed network connection
Jul 9 02:12:31.109064390 (377h2m5.879705725s) [seg_pos={0 0}, upload=301625]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:12:31.339469240 (377h2m6.110110565s) [upload=301625]: Done uploading segments...
Jul 9 02:12:31.339495170 (377h2m6.110136505s) [upload=301625]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.871991251s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.......X......X..........X.............X...............X...X] duration:
1m0.234708808s
FAIL (2025-07-09 02:13:07 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
.
Jul 9 02:12:07.627735816 (377h1m42.398377151s) [upload=301626]: Got next segment. Inline: false
Jul 9 02:12:07.627768629 (377h1m42.398409964s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:12:07.627773568 (377h1m42.398414903s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:12:08.075105689 (377h1m42.845747024s) [seg_pos={0 0}, upload=301626]: Begin upload segment...
Jul 9 02:12:08.075172738 (377h1m42.845814073s) [seg_pos={0 0}, upload=301626]: Done begin upload segment.
Jul 9 02:12:08.075200341 (377h1m42.845841676s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075226711 (377h1m42.845868046s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075642747 (377h1m42.846284102s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076099570 (377h1m42.846740905s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076193791 (377h1m42.846835126s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076453428 (377h1m42.847094763s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076686334 (377h1m42.847327669s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076926883 (377h1m42.847568218s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076956250 (377h1m42.847597585s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077181110 (377h1m42.847822445s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077198733 (377h1m42.847840068s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077433213 (377h1m42.848074548s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077452068 (377h1m42.848093403s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077655618 (377h1m42.848296953s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077884866 (377h1m42.848526201s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078100028 (377h1m42.848741363s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078310551 (377h1m42.848951886s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078506095 (377h1m42.84914743s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:09.631539904 (377h1m44.402181239s) [upload=301626]: Next returned nil segment
Jul 9 02:12:09.631556275 (377h1m44.4021976s) [upload=301626]: Waiting for error group managing segments...
Jul 9 02:12:10.026556908 (377h1m44.797198243s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.029581435 (377h1m44.80022277s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.030348031 (377h1m44.800989366s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.037300182 (377h1m44.807941517s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.038590761 (377h1m44.809232096s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.040614022 (377h1m44.811255347s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.243165042 (377h1m45.013806377s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.294115297 (377h1m45.064756632s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:12.752127764 (377h1m47.522769099s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:29.819338082 (377h2m4.589979417s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:35.349456003 (377h2m10.120097338s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:40.626537094 (377h2m15.397178419s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:42.766847833 (377h2m17.537489168s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:13:06.981429964 (377h2m41.752071299s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:15mgZMkU28J1JwrRd4XbvZFyjp8G7CTBkJCxbzUYqKeSeqCZ9F, address:104.36.82.7:4944): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981472496 (377h2m41.752113821s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; EOF
Jul 9 02:13:06.981498496 (377h2m41.752139821s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:12pHU3Xp1sDgnmYMcbWZD2DivxdEUzqnBVJaiRaQipuRy2d75u5, address:104.36.82.18:4375): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981567808 (377h2m41.752209143s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981595070 (377h2m41.752236395s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection
Jul 9 02:13:06.981611983 (377h2m41.752253318s) [seg_pos={0 0}, upload=301626]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:13:07.214001409 (377h2m41.984642744s) [upload=301626]: Done uploading segments...
Jul 9 02:13:07.214031306 (377h2m41.984672641s) [upload=301626]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.234731382s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [...XX.XX..X......XXX....XX.....XX...X.XX........X.X......X..] duration:
44.886826972s
FAIL (2025-07-09 02:13:01 UTC)
orbiter warning: context deadline exceeded
Duration: 44.886845598s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [......................X.................X......X.........XXX] duration:
30.232064116s
FAIL (2025-07-09 02:12:31 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
71 (377h1m36.523589407s) [upload=301625]: Uploading segments...
Jul 9 02:12:01.753019818 (377h1m36.523661153s) [upload=301625]: Got next segment. Inline: false
Jul 9 02:12:01.753045758 (377h1m36.523687093s) [upload=301625]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:12:01.753050808 (377h1m36.523692143s) [upload=301625]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:12:01.989838026 (377h1m36.760479361s) [seg_pos={0 0}, upload=301625]: Begin upload segment...
Jul 9 02:12:01.989881920 (377h1m36.760523255s) [seg_pos={0 0}, upload=301625]: Done begin upload segment.
Jul 9 02:12:01.989897190 (377h1m36.760538525s) [noise=true , address=45.91.116.14:4191 , piece_id=C5GROPMRE7AYIG4SDMVDH63VJUKCYMW6U7EKM3MVIOUZBC6DUQMQ, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.989905004 (377h1m36.760546349s) [noise=true , address=45.91.116.7:4011 , piece_id=7H4X3T745B5KFEQ4LJGOLBFDKRJY7WAMKA2P3UT3GUHPJQ7EVPGQ, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.990218293 (377h1m36.760859629s) [noise=true , address=77.90.31.16:4734 , piece_id=K4LSJTW4FHH3HURFBIB7OONWE32CZGTO636ESXIWTNVCDOXEM3CA, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.990226188 (377h1m36.760867523s) [noise=true , address=45.91.116.5:4297 , piece_id=AJZOVXX5X25H75ESLPFCPXA57JLKZJWJS67MIHLP6XWNNABXEURA, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.990459024 (377h1m36.761100359s) [noise=true , address=45.91.116.6:4974 , piece_id=CWZNS2UQYFSKCAAY2IA55TAU5DQV6P7I5TJSA4IULP5V4KJJ6TBQ, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.990525110 (377h1m36.761166445s) [noise=true , address=104.36.82.13:4186 , piece_id=7GMI3HN2WNK6CRLORWPCUMFSF5URMWE7OZV5AGBQBUTCTJ5TMD7Q, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.990686329 (377h1m36.761327664s) [noise=true , address=45.91.116.11:4119 , piece_id=D7SQQI3L6XV7QDSUGUXU6B2KA5SQW5VN7ZU2LWVKZSBI2ENQ46YQ, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.990920928 (377h1m36.761562263s) [noise=true , address=45.91.116.8:4045 , piece_id=RJ6ECPUGUOXRJCCAZWZ2UZSDA4XC7QWTPLVA2M435VGP6ZFVQR4A, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.991140709 (377h1m36.761782044s) [noise=true , address=104.36.82.2:4730 , piece_id=DLZH3HN2DJ4RG7IBONW5IRKJFGQBSYZCBEX6H32KR5TTFPIKHEXQ, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.991363214 (377h1m36.762004549s) [noise=true , address=77.90.31.17:4772 , piece_id=3AP77S5YRSLMEEWS4KIYYJ6BXYVIEW5ESRFXVZIS4IL5LT77TV2Q, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.991580861 (377h1m36.762222196s) [noise=true , address=45.91.116.10:4082 , piece_id=R7RFWF454POA57PH4QJDNOR4CDRSVWKE4GLNSAUQIRH6YBBQBOLQ, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.991778940 (377h1m36.762420275s) [noise=true , address=45.91.116.13:4155 , piece_id=MP7WL4IKLOQK53RSTK2SKYKIAUK5DDKVBEWRLFKJBAQGTWQMFM7A, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.992034609 (377h1m36.762675944s) [noise=true , address=104.36.82.12:4150 , piece_id=5AFHJ72DS2GJYUNACDBG4SRZPYHUGOUSEAJNWKH2362UE2S6KZ5A, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.992236064 (377h1m36.762877399s) [noise=true , address=45.91.116.13:4153 , piece_id=GZX6EVLDMA4LI436AIHADCESLSYPNBXW6S4QBOXJMDSVSFM4KKHA, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.993996933 (377h1m36.764638268s) [noise=true , address=104.36.82.2:4731 , piece_id=CIWRKY5Q6PSNPQZVOXBRJ2WCDSKV2VB2YXYPMN4CVKA7POC3CXKQ, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:01.994263944 (377h1m36.764905279s) [noise=true , address=104.36.82.14:4222 , piece_id=3NTL5ZFBRX2JSPOGHDURULC3X7IMWSMJK52MW7MJ4QF3YB7OJW4A, seg_pos={0 0}, upload=301625]: Uploading piece...
Jul 9 02:12:03.750924051 (377h1m38.521565386s) [upload=301625]: Next returned nil segment
Jul 9 02:12:03.750940513 (377h1m38.521581848s) [upload=301625]: Waiting for error group managing segments...
Jul 9 02:12:04.047569006 (377h1m38.818210341s) [noise=true , address=45.91.116.10:4082 , piece_id=R7RFWF454POA57PH4QJDNOR4CDRSVWKE4GLNSAUQIRH6YBBQBOLQ, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:04.049842796 (377h1m38.820484131s) [noise=true , address=45.91.116.13:4153 , piece_id=GZX6EVLDMA4LI436AIHADCESLSYPNBXW6S4QBOXJMDSVSFM4KKHA, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:04.050277458 (377h1m38.820918793s) [noise=true , address=45.91.116.14:4191 , piece_id=C5GROPMRE7AYIG4SDMVDH63VJUKCYMW6U7EKM3MVIOUZBC6DUQMQ, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:04.050419810 (377h1m38.821061145s) [noise=true , address=45.91.116.7:4011 , piece_id=7H4X3T745B5KFEQ4LJGOLBFDKRJY7WAMKA2P3UT3GUHPJQ7EVPGQ, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:04.054000681 (377h1m38.824642006s) [noise=true , address=45.91.116.5:4297 , piece_id=AJZOVXX5X25H75ESLPFCPXA57JLKZJWJS67MIHLP6XWNNABXEURA, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:04.058369419 (377h1m38.829010754s) [noise=true , address=45.91.116.13:4155 , piece_id=MP7WL4IKLOQK53RSTK2SKYKIAUK5DDKVBEWRLFKJBAQGTWQMFM7A, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:04.063800109 (377h1m38.834441444s) [noise=true , address=45.91.116.11:4119 , piece_id=D7SQQI3L6XV7QDSUGUXU6B2KA5SQW5VN7ZU2LWVKZSBI2ENQ46YQ, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:04.079959624 (377h1m38.850600959s) [noise=true , address=77.90.31.17:4772 , piece_id=3AP77S5YRSLMEEWS4KIYYJ6BXYVIEW5ESRFXVZIS4IL5LT77TV2Q, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:04.094478832 (377h1m38.865120167s) [noise=true , address=77.90.31.16:4734 , piece_id=K4LSJTW4FHH3HURFBIB7OONWE32CZGTO636ESXIWTNVCDOXEM3CA, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:04.245456283 (377h1m39.016097618s) [noise=true , address=45.91.116.6:4974 , piece_id=CWZNS2UQYFSKCAAY2IA55TAU5DQV6P7I5TJSA4IULP5V4KJJ6TBQ, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:05.376301634 (377h1m40.146942969s) [noise=true , address=45.91.116.8:4045 , piece_id=RJ6ECPUGUOXRJCCAZWZ2UZSDA4XC7QWTPLVA2M435VGP6ZFVQR4A, seg_pos={0 0}, upload=301625]: Done uploading piece. err: <nil>
Jul 9 02:12:31.108895367 (377h2m5.879536702s) [noise=true , address=104.36.82.12:4150 , piece_id=5AFHJ72DS2GJYUNACDBG4SRZPYHUGOUSEAJNWKH2362UE2S6KZ5A, seg_pos={0 0}, upload=301625]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:43048->104.36.82.12:4150: use of closed network connection; write tcp 188.166.225.170:43048->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43048->104.36.82.12:4150: use of closed network connection
Jul 9 02:12:31.108946284 (377h2m5.879587619s) [noise=true , address=104.36.82.2:4730 , piece_id=DLZH3HN2DJ4RG7IBONW5IRKJFGQBSYZCBEX6H32KR5TTFPIKHEXQ, seg_pos={0 0}, upload=301625]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:43754->104.36.82.2:4730: use of closed network connection; write tcp 188.166.225.170:43754->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43754->104.36.82.2:4730: use of closed network connection
Jul 9 02:12:31.108983765 (377h2m5.879625101s) [noise=true , address=104.36.82.14:4222 , piece_id=3NTL5ZFBRX2JSPOGHDURULC3X7IMWSMJK52MW7MJ4QF3YB7OJW4A, seg_pos={0 0}, upload=301625]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:37012->104.36.82.14:4222: use of closed network connection; write tcp 188.166.225.170:37012->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37012->104.36.82.14:4222: use of closed network connection
Jul 9 02:12:31.109017410 (377h2m5.879658745s) [noise=true , address=104.36.82.2:4731 , piece_id=CIWRKY5Q6PSNPQZVOXBRJ2WCDSKV2VB2YXYPMN4CVKA7POC3CXKQ, seg_pos={0 0}, upload=301625]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:54750->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:54750->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54750->104.36.82.2:4731: use of closed network connection
Jul 9 02:12:31.109051516 (377h2m5.879692841s) [noise=true , address=104.36.82.13:4186 , piece_id=7GMI3HN2WNK6CRLORWPCUMFSF5URMWE7OZV5AGBQBUTCTJ5TMD7Q, seg_pos={0 0}, upload=301625]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 188.166.225.170:33706->104.36.82.13:4186: use of closed network connection; write tcp 188.166.225.170:33706->104.36.82.13:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33706->104.36.82.13:4186: use of closed network connection
Jul 9 02:12:31.109064390 (377h2m5.879705725s) [seg_pos={0 0}, upload=301625]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:12:31.339469240 (377h2m6.110110565s) [upload=301625]: Done uploading segments...
Jul 9 02:12:31.339495170 (377h2m6.110136505s) [upload=301625]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.232100877s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [..X..X............X......X..X...X.X.X.X...X...............X.] duration:
20.548400915s
FAIL (2025-07-09 02:12:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 20.548420091s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [....XX.XX..X......XXX....XX.....XX...X.XX........X.X......X.] duration:
39.646061717s
FAIL (2025-07-09 02:12:16 UTC)
orbiter warning: context deadline exceeded
Duration: 39.64610001s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [........X......X..........X.............X...............X...] duration:
42.74268599s
FAIL (2025-07-09 02:11:49 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
7177 (376h58m5.878188512s) [upload=301581]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:08:31.554710025 (376h58m6.32535136s) [seg_pos={0 0}, upload=301581]: Begin upload segment...
Jul 9 02:08:31.554766423 (376h58m6.325407758s) [seg_pos={0 0}, upload=301581]: Done begin upload segment.
Jul 9 02:08:31.554787664 (376h58m6.325428989s) [noise=true , address=104.36.82.40:4944 , piece_id=2DMADEN45IW5WXG6SSK5Y72S2DKZBDBY2NXRZBHWYIICLKWDZFEQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.554824724 (376h58m6.325466059s) [noise=true , address=77.90.31.12:4614 , piece_id=5TIDDQKRKPQE3GJ6CMJJZTU5E3LYRTXMMS63G7EAV64Q6AXFWQRA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.555170586 (376h58m6.325811921s) [noise=true , address=45.91.116.9:4266 , piece_id=4TFY2OQQ472NIZV4LUREV2BLF75RITUK3OEF43W3KURP2AZYRGCQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.555483365 (376h58m6.3261247s) [noise=true , address=104.36.82.44:4889 , piece_id=SMJJSN3XNTE53GM357QFJMOR2L6EUZP5VG5LH2HLYVG7MFUZJ7AA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.555790893 (376h58m6.326432218s) [noise=true , address=45.91.116.7:4041 , piece_id=DI4N5ADKDSKFS6D5HUHXONI63SKBM2BLLRDNTCRD5RWSSKS2W3IQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.556074445 (376h58m6.32671578s) [noise=true , address=77.90.31.17:4780 , piece_id=6XI3ESZNONU2SZJJ3ELUMOMZKSOQY4SBOBRPRBERSTZ2AZRKX57A, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.556202771 (376h58m6.326844106s) [noise=true , address=193.149.164.143:4693 , piece_id=ZJKXKNFUYTUKNW25NI5IY5GKBQYNHTAKN6VRLBMYTAMCYKVYXMHQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.556569773 (376h58m6.327211108s) [noise=true , address=104.36.82.20:4441 , piece_id=XSH7ISLZ367ZCC6XM4YN7DDRERMG3MEBFRNH5K6WLXTKIIIPO6SA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.556904713 (376h58m6.327546048s) [noise=true , address=77.90.31.21:4886 , piece_id=MLVSZ27JZLYNG23IRDFTDMXJRTZ3IMHUOPDGDPOO42GCERLZUQOA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.557071302 (376h58m6.327712637s) [noise=true , address=104.36.82.35:4590 , piece_id=7KGB363U5XFEOK3UJKJXJ57P7T6XBMDUI4UZKQQ3Q4XJ3TJV54XQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.557342130 (376h58m6.327983465s) [noise=true , address=104.36.82.17:4345 , piece_id=JEG2KMNQ4ZXAYERB65ATCCC6FAYZX5S57GEM7YPV4274LI636YSA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.557595104 (376h58m6.328236439s) [noise=true , address=77.90.31.10:4528 , piece_id=7R3R7QEQIQNYBV7BJF3ZGCMVJFH6OMIXZQTU3KCPSJGQTVT37RRA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.557906210 (376h58m6.328547545s) [noise=true , address=77.90.31.16:4739 , piece_id=A52Q7XZMXUFM5BTN6XYQUAML4HVR55D6IIAY6J7P3OCFTHRIXTVQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.558016280 (376h58m6.328657615s) [noise=true , address=104.36.82.23:4469 , piece_id=N235KELWJET4DXCIBYJGRMR4U5GFVMERXTDM3BM62KR7WBCPJJPA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.558293511 (376h58m6.328934846s) [noise=true , address=104.36.82.41:4783 , piece_id=L73QGLG7NQJR7VC232MVNMZXZWUL75VEISACRGNMH32GT56BFBBQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.558573757 (376h58m6.329215082s) [noise=true , address=104.36.82.2:4757 , piece_id=MGSUR22FTPRP6A7E7DEZNQBFBPOYQBNTGT5ZNEM64FQTCPAAJCTQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.558925379 (376h58m6.329566724s) [noise=true , address=77.90.31.11:4588 , piece_id=HMT3XDYW2QZCZJDKLXV7ISS3JTI7OSFC3HMRKFCZO5FBUH4TP6NA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.559409055 (376h58m6.33005039s) [noise=true , address=79.127.201.214:4305 , piece_id=OL3IFYRAUOVBYLWGO4QXOMDZFGQTCXZAG73F34M7SDSLZ46MNCCA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:32.571736414 (376h58m7.342377739s) [noise=true , address=79.127.201.214:4305 , piece_id=OL3IFYRAUOVBYLWGO4QXOMDZFGQTCXZAG73F34M7SDSLZ46MNCCA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:12HLYBv5M6NE1Njgfz4WdagyLUqPJyc9WjEgr1CfVVEcso7HnCX, address:79.127.201.214:4305): protocol: storage node overloaded, request limit: 2000
Jul 9 02:08:32.707702667 (376h58m7.478344003s) [upload=301581]: Next returned nil segment
Jul 9 02:08:32.707717586 (376h58m7.478358922s) [upload=301581]: Waiting for error group managing segments...
Jul 9 02:08:33.045788856 (376h58m7.816430191s) [noise=true , address=193.149.164.143:4693 , piece_id=ZJKXKNFUYTUKNW25NI5IY5GKBQYNHTAKN6VRLBMYTAMCYKVYXMHQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.477787513 (376h58m8.248428848s) [noise=true , address=45.91.116.7:4041 , piece_id=DI4N5ADKDSKFS6D5HUHXONI63SKBM2BLLRDNTCRD5RWSSKS2W3IQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.492837816 (376h58m8.263479151s) [noise=true , address=77.90.31.17:4780 , piece_id=6XI3ESZNONU2SZJJ3ELUMOMZKSOQY4SBOBRPRBERSTZ2AZRKX57A, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.505858116 (376h58m8.276499451s) [noise=true , address=77.90.31.10:4528 , piece_id=7R3R7QEQIQNYBV7BJF3ZGCMVJFH6OMIXZQTU3KCPSJGQTVT37RRA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.510054294 (376h58m8.280695629s) [noise=true , address=77.90.31.12:4614 , piece_id=5TIDDQKRKPQE3GJ6CMJJZTU5E3LYRTXMMS63G7EAV64Q6AXFWQRA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.511325926 (376h58m8.281967261s) [noise=true , address=77.90.31.11:4588 , piece_id=HMT3XDYW2QZCZJDKLXV7ISS3JTI7OSFC3HMRKFCZO5FBUH4TP6NA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.526917155 (376h58m8.29755849s) [noise=true , address=77.90.31.21:4886 , piece_id=MLVSZ27JZLYNG23IRDFTDMXJRTZ3IMHUOPDGDPOO42GCERLZUQOA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.538345038 (376h58m8.308986373s) [noise=true , address=77.90.31.16:4739 , piece_id=A52Q7XZMXUFM5BTN6XYQUAML4HVR55D6IIAY6J7P3OCFTHRIXTVQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.609285488 (376h58m8.379926823s) [noise=true , address=45.91.116.9:4266 , piece_id=4TFY2OQQ472NIZV4LUREV2BLF75RITUK3OEF43W3KURP2AZYRGCQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:46.984505697 (376h58m21.755147032s) [noise=true , address=104.36.82.20:4441 , piece_id=XSH7ISLZ367ZCC6XM4YN7DDRERMG3MEBFRNH5K6WLXTKIIIPO6SA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:52.339907914 (376h58m27.110549249s) [noise=true , address=104.36.82.41:4783 , piece_id=L73QGLG7NQJR7VC232MVNMZXZWUL75VEISACRGNMH32GT56BFBBQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:09:01.032804568 (376h58m35.803445903s) [noise=true , address=104.36.82.44:4889 , piece_id=SMJJSN3XNTE53GM357QFJMOR2L6EUZP5VG5LH2HLYVG7MFUZJ7AA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:09:08.673338691 (376h58m43.443980026s) [noise=true , address=104.36.82.35:4590 , piece_id=7KGB363U5XFEOK3UJKJXJ57P7T6XBMDUI4UZKQQ3Q4XJ3TJV54XQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:09:30.410919586 (376h59m5.181560921s) [noise=true , address=104.36.82.2:4757 , piece_id=MGSUR22FTPRP6A7E7DEZNQBFBPOYQBNTGT5ZNEM64FQTCPAAJCTQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:104.36.82.2:4757): protocol: expected piece hash; write tcp 188.166.225.170:43178->104.36.82.2:4757: use of closed network connection; write tcp 188.166.225.170:43178->104.36.82.2:4757: use of closed network connection; EOF
Jul 9 02:09:30.410974972 (376h59m5.181616307s) [noise=true , address=104.36.82.40:4944 , piece_id=2DMADEN45IW5WXG6SSK5Y72S2DKZBDBY2NXRZBHWYIICLKWDZFEQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:12KMFsxzp7SfMiFBuMTaWGVyJoRTMw7ZRcfEpPDv1BmbViGF6K9, address:104.36.82.40:4944): protocol: write tcp 188.166.225.170:54250->104.36.82.40:4944: use of closed network connection; write tcp 188.166.225.170:54250->104.36.82.40:4944: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54250->104.36.82.40:4944: use of closed network connection
Jul 9 02:09:30.411014277 (376h59m5.181655612s) [noise=true , address=104.36.82.23:4469 , piece_id=N235KELWJET4DXCIBYJGRMR4U5GFVMERXTDM3BM62KR7WBCPJJPA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:09:30.411035838 (376h59m5.181677174s) [noise=true , address=104.36.82.17:4345 , piece_id=JEG2KMNQ4ZXAYERB65ATCCC6FAYZX5S57GEM7YPV4274LI636YSA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:1ro6Z6UAC6dNRPRWMjdMTTjhg1CvmWPENs7JcqFHxHMpmz5Fnv, address:104.36.82.17:4345): protocol: write tcp 188.166.225.170:41288->104.36.82.17:4345: use of closed network connection; write tcp 188.166.225.170:41288->104.36.82.17:4345: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41288->104.36.82.17:4345: use of closed network connection
Jul 9 02:09:30.411049184 (376h59m5.181690519s) [seg_pos={0 0}, upload=301581]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:09:30.640189311 (376h59m5.410830686s) [upload=301581]: Done uploading segments...
Jul 9 02:09:30.640217886 (376h59m5.410859221s) [upload=301581]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 42.74270098s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [...X..X............X......X..X...X.X.X.X...X...............X] duration:
30.001394385s
FAIL (2025-07-09 02:11:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001406439s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [.....XX.XX..X......XXX....XX.....XX...X.XX........X.X......X] duration:
1m0.000945225s
FAIL (2025-07-09 02:11:36 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.000958231s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [X......................X.................X......X.........XX] duration:
30.232249023s
FAIL (2025-07-09 02:11:23 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
.7456104s) [upload=301604]: Done uploading segments...
Jul 9 02:10:52.974995766 (377h0m27.745637091s) [upload=301604]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Jul 9 02:10:53.845098218 (377h0m28.615739553s) [upload=301612]: Starting upload
Jul 9 02:10:53.845214179 (377h0m28.615855554s) [upload=301612]: Done starting upload
Jul 9 02:10:53.845350420 (377h0m28.615991755s) [upload=301612]: Uploading segments...
Jul 9 02:10:53.845487382 (377h0m28.616128707s) [upload=301612]: Got next segment. Inline: false
Jul 9 02:10:53.845528581 (377h0m28.616169916s) [upload=301612]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:10:53.845533891 (377h0m28.616175216s) [upload=301612]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:10:54.082701577 (377h0m28.853342962s) [seg_pos={0 0}, upload=301612]: Begin upload segment...
Jul 9 02:10:54.082755410 (377h0m28.853396745s) [seg_pos={0 0}, upload=301612]: Done begin upload segment.
Jul 9 02:10:54.082779396 (377h0m28.853420731s) [noise=true , address=45.91.116.11:4117 , piece_id=SMF3HTML2E2BUUILFZQ5IMUO7IZ24LJBU5MG3N43X7JRJSGHPKFA, seg_pos={0 0}, upload=301612]: Uploading piece...
Jul 9 02:10:54.082789314 (377h0m28.85343066s) [noise=true , address=104.36.82.9:4007 , piece_id=GLB6FGEXUDR4IRRKL7IRBR3NFPWRB6LR6EOASTNDZWHX3YKHPL6A, seg_pos={0 0}, upload=301612]: Uploading piece...
Jul 9 02:10:54.083402528 (377h0m28.854043934s) [noise=true , address=77.90.31.21:4877 , piece_id=5TW3M252BEKGCHM3WSFKGRNSXFIBBOIJOFOTQCF34NIJO2FOWWOA, seg_pos={0 0}, upload=301612]: Uploading piece...
Jul 9 02:10:54.084664232 (377h0m28.855305567s) [noise=true , address=77.90.31.19:4842 , piece_id=CJZENK7SZOEL4U3PEBG6KQVRFHMBOBECCEK7OXR3DCK5JKFHQA3A, seg_pos={0 0}, upload=301612]: Uploading piece...
Jul 9 02:10:54.085275231 (377h0m28.855916576s) [noise=true , address=45.91.116.6:4975 , piece_id=BURKUUCQKEOYR3IJDEV2TBQ4YCM6M3PMTWSBMS5U5FG4G5E4ZDTQ, seg_pos={0 0}, upload=301612]: Uploading piece...
Jul 9 02:10:54.085671409 (377h0m28.856312744s) [noise=true , address=45.91.116.10:4083 , piece_id=SMCJ6YQAIHXOL32DDEAQS5UL5SMIR3WYNX4CLAB4KWMBEK7AGH4Q, seg_pos={0 0}, upload=301612]: Uploading piece...
Jul 9 02:10:54.085729951 (377h0m28.856371286s) [noise=true , address=104.36.82.4:4804 , piece_id=6HXXXR7H3GHILZ4ZAY3VGG7DZ2LBDFBKLE75W2WBGKNWMEALIWQA, seg_pos={0 0}, upload=301612]: Uploading piece...
Jul 9 02:10:54.086252140 (377h0m28.856893475s) [noise=true , address=45.91.116.7:4009 , piece_id=FBN3VGOLRYBZCGUZQ4JMP5ROFVWCBIN2TNASMVMVIPR7FQBCCSRA, seg_pos={0 0}, upload=301612]: Uploading piece...
Jul 9 02:10:54.086845194 (377h0m28.857486529s) [noise=true , address=104.36.82.12:4151 , piece_id=LTRMKWC6MVIWIBE4VJCONZKRWV5GB6TWU6HL67BM6IIZFVQYKDGA, seg_pos={0 0}, upload=301612]: Uploading piece...
Jul 9 02:10:54.087960258 (377h0m28.858601653s) [noise=true , address=45.91.116.14:4190 , piece_id=UHRSIM56MIQCYX4YLDSOWWY4LT2UIPYTLDQ5R47KKA6OCEQTJUVA, seg_pos={0 0}, upload=301612]: Uploading piece...
Jul 9 02:10:54.091001476 (377h0m28.861642811s) [noise=true , address=104.36.82.14:4223 , piece_id=4RC42Y67WLTUWKKBVE4E2XBLWUNZZCVGPRFBWBFPNJ76DEHFFOQA, seg_pos={0 0}, upload=301612]: Uploading piece...
Jul 9 02:10:54.092378100 (377h0m28.863019435s) [noise=true , address=45.91.116.8:4045 , piece_id=N5XZZ7SNR4Z7AJUPQ4VDRQXD7KTKRR4ZCNG4TOG3SCSFIGJVX3KA, seg_pos={0 0}, upload=301612]: Uploading piece...
Jul 9 02:10:54.093642638 (377h0m28.864283973s) [noise=true , address=104.36.82.11:4114 , piece_id=UMK2NAG45ZWDRFPHZ7A2IKZXCYDGG6ON3GFZG7OAVLVQMJSZYW6Q, seg_pos={0 0}, upload=301612]: Uploading piece...
Jul 9 02:10:54.094779695 (377h0m28.86542103s) [noise=true , address=104.36.82.8:4972 , piece_id=HG6M6C3XL62GHNQPWRKIVRBWWGW7A733IBETKMPQQPROUEDN5NMQ, seg_pos={0 0}, upload=301612]: Uploading piece...
Jul 9 02:10:54.095239033 (377h0m28.865880368s) [noise=true , address=77.90.31.12:4592 , piece_id=WFXWYBHYDHGZRYE2VSI4FM6TYX7HBAKJ3N3S6J4S2CTPABN65GBQ, seg_pos={0 0}, upload=301612]: Uploading piece...
Jul 9 02:10:54.096059473 (377h0m28.866700818s) [noise=true , address=77.90.31.17:4772 , piece_id=6DQV2GSLDYWZ3QFOJNGIPN5EXUWOMJ73S5X7AVTESU7NATLVUCKA, seg_pos={0 0}, upload=301612]: Uploading piece...
Jul 9 02:10:55.726431962 (377h0m30.497073287s) [upload=301612]: Next returned nil segment
Jul 9 02:10:55.726448595 (377h0m30.49708993s) [upload=301612]: Waiting for error group managing segments...
Jul 9 02:10:56.007649884 (377h0m30.778291219s) [noise=true , address=104.36.82.9:4007 , piece_id=GLB6FGEXUDR4IRRKL7IRBR3NFPWRB6LR6EOASTNDZWHX3YKHPL6A, seg_pos={0 0}, upload=301612]: Done uploading piece. err: <nil>
Jul 9 02:10:56.139405692 (377h0m30.910047027s) [noise=true , address=45.91.116.14:4190 , piece_id=UHRSIM56MIQCYX4YLDSOWWY4LT2UIPYTLDQ5R47KKA6OCEQTJUVA, seg_pos={0 0}, upload=301612]: Done uploading piece. err: <nil>
Jul 9 02:10:56.156511178 (377h0m30.927152513s) [noise=true , address=45.91.116.6:4975 , piece_id=BURKUUCQKEOYR3IJDEV2TBQ4YCM6M3PMTWSBMS5U5FG4G5E4ZDTQ, seg_pos={0 0}, upload=301612]: Done uploading piece. err: <nil>
Jul 9 02:10:56.171571860 (377h0m30.942213195s) [noise=true , address=77.90.31.21:4877 , piece_id=5TW3M252BEKGCHM3WSFKGRNSXFIBBOIJOFOTQCF34NIJO2FOWWOA, seg_pos={0 0}, upload=301612]: Done uploading piece. err: <nil>
Jul 9 02:10:56.172286658 (377h0m30.942927993s) [noise=true , address=77.90.31.17:4772 , piece_id=6DQV2GSLDYWZ3QFOJNGIPN5EXUWOMJ73S5X7AVTESU7NATLVUCKA, seg_pos={0 0}, upload=301612]: Done uploading piece. err: <nil>
Jul 9 02:10:56.178119216 (377h0m30.948760551s) [noise=true , address=77.90.31.19:4842 , piece_id=CJZENK7SZOEL4U3PEBG6KQVRFHMBOBECCEK7OXR3DCK5JKFHQA3A, seg_pos={0 0}, upload=301612]: Done uploading piece. err: <nil>
Jul 9 02:10:56.180567591 (377h0m30.951208926s) [noise=true , address=45.91.116.11:4117 , piece_id=SMF3HTML2E2BUUILFZQ5IMUO7IZ24LJBU5MG3N43X7JRJSGHPKFA, seg_pos={0 0}, upload=301612]: Done uploading piece. err: <nil>
Jul 9 02:10:56.191668529 (377h0m30.962309904s) [noise=true , address=77.90.31.12:4592 , piece_id=WFXWYBHYDHGZRYE2VSI4FM6TYX7HBAKJ3N3S6J4S2CTPABN65GBQ, seg_pos={0 0}, upload=301612]: Done uploading piece. err: <nil>
Jul 9 02:10:56.216758580 (377h0m30.987399915s) [noise=true , address=45.91.116.7:4009 , piece_id=FBN3VGOLRYBZCGUZQ4JMP5ROFVWCBIN2TNASMVMVIPR7FQBCCSRA, seg_pos={0 0}, upload=301612]: Done uploading piece. err: <nil>
Jul 9 02:10:56.232131481 (377h0m31.002772826s) [noise=true , address=45.91.116.8:4045 , piece_id=N5XZZ7SNR4Z7AJUPQ4VDRQXD7KTKRR4ZCNG4TOG3SCSFIGJVX3KA, seg_pos={0 0}, upload=301612]: Done uploading piece. err: <nil>
Jul 9 02:10:56.238338475 (377h0m31.0089798s) [noise=true , address=45.91.116.10:4083 , piece_id=SMCJ6YQAIHXOL32DDEAQS5UL5SMIR3WYNX4CLAB4KWMBEK7AGH4Q, seg_pos={0 0}, upload=301612]: Done uploading piece. err: <nil>
Jul 9 02:11:11.580188804 (377h0m46.350830139s) [noise=true , address=104.36.82.4:4804 , piece_id=6HXXXR7H3GHILZ4ZAY3VGG7DZ2LBDFBKLE75W2WBGKNWMEALIWQA, seg_pos={0 0}, upload=301612]: Done uploading piece. err: <nil>
Jul 9 02:11:23.199330306 (377h0m57.969971641s) [noise=true , address=104.36.82.14:4223 , piece_id=4RC42Y67WLTUWKKBVE4E2XBLWUNZZCVGPRFBWBFPNJ76DEHFFOQA, seg_pos={0 0}, upload=301612]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:52650->104.36.82.14:4223: use of closed network connection; write tcp 188.166.225.170:52650->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52650->104.36.82.14:4223: use of closed network connection
Jul 9 02:11:23.199380472 (377h0m57.970021807s) [noise=true , address=104.36.82.12:4151 , piece_id=LTRMKWC6MVIWIBE4VJCONZKRWV5GB6TWU6HL67BM6IIZFVQYKDGA, seg_pos={0 0}, upload=301612]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:56800->104.36.82.12:4151: use of closed network connection; write tcp 188.166.225.170:56800->104.36.82.12:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56800->104.36.82.12:4151: use of closed network connection
Jul 9 02:11:23.199416490 (377h0m57.970057825s) [noise=true , address=104.36.82.11:4114 , piece_id=UMK2NAG45ZWDRFPHZ7A2IKZXCYDGG6ON3GFZG7OAVLVQMJSZYW6Q, seg_pos={0 0}, upload=301612]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:46036->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:46036->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46036->104.36.82.11:4114: use of closed network connection
Jul 9 02:11:23.199448892 (377h0m57.970090217s) [noise=true , address=104.36.82.8:4972 , piece_id=HG6M6C3XL62GHNQPWRKIVRBWWGW7A733IBETKMPQQPROUEDN5NMQ, seg_pos={0 0}, upload=301612]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:35608->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:35608->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35608->104.36.82.8:4972: use of closed network connection
Jul 9 02:11:23.199463290 (377h0m57.970104625s) [seg_pos={0 0}, upload=301612]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:11:23.430070365 (377h0m58.20071169s) [upload=301612]: Done uploading segments...
Jul 9 02:11:23.430093970 (377h0m58.200735305s) [upload=301612]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.232259733s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.........X......X..........X.............X...............X..] duration:
42.68457557s
FAIL (2025-07-09 02:11:06 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
7177 (376h58m5.878188512s) [upload=301581]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:08:31.554710025 (376h58m6.32535136s) [seg_pos={0 0}, upload=301581]: Begin upload segment...
Jul 9 02:08:31.554766423 (376h58m6.325407758s) [seg_pos={0 0}, upload=301581]: Done begin upload segment.
Jul 9 02:08:31.554787664 (376h58m6.325428989s) [noise=true , address=104.36.82.40:4944 , piece_id=2DMADEN45IW5WXG6SSK5Y72S2DKZBDBY2NXRZBHWYIICLKWDZFEQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.554824724 (376h58m6.325466059s) [noise=true , address=77.90.31.12:4614 , piece_id=5TIDDQKRKPQE3GJ6CMJJZTU5E3LYRTXMMS63G7EAV64Q6AXFWQRA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.555170586 (376h58m6.325811921s) [noise=true , address=45.91.116.9:4266 , piece_id=4TFY2OQQ472NIZV4LUREV2BLF75RITUK3OEF43W3KURP2AZYRGCQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.555483365 (376h58m6.3261247s) [noise=true , address=104.36.82.44:4889 , piece_id=SMJJSN3XNTE53GM357QFJMOR2L6EUZP5VG5LH2HLYVG7MFUZJ7AA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.555790893 (376h58m6.326432218s) [noise=true , address=45.91.116.7:4041 , piece_id=DI4N5ADKDSKFS6D5HUHXONI63SKBM2BLLRDNTCRD5RWSSKS2W3IQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.556074445 (376h58m6.32671578s) [noise=true , address=77.90.31.17:4780 , piece_id=6XI3ESZNONU2SZJJ3ELUMOMZKSOQY4SBOBRPRBERSTZ2AZRKX57A, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.556202771 (376h58m6.326844106s) [noise=true , address=193.149.164.143:4693 , piece_id=ZJKXKNFUYTUKNW25NI5IY5GKBQYNHTAKN6VRLBMYTAMCYKVYXMHQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.556569773 (376h58m6.327211108s) [noise=true , address=104.36.82.20:4441 , piece_id=XSH7ISLZ367ZCC6XM4YN7DDRERMG3MEBFRNH5K6WLXTKIIIPO6SA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.556904713 (376h58m6.327546048s) [noise=true , address=77.90.31.21:4886 , piece_id=MLVSZ27JZLYNG23IRDFTDMXJRTZ3IMHUOPDGDPOO42GCERLZUQOA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.557071302 (376h58m6.327712637s) [noise=true , address=104.36.82.35:4590 , piece_id=7KGB363U5XFEOK3UJKJXJ57P7T6XBMDUI4UZKQQ3Q4XJ3TJV54XQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.557342130 (376h58m6.327983465s) [noise=true , address=104.36.82.17:4345 , piece_id=JEG2KMNQ4ZXAYERB65ATCCC6FAYZX5S57GEM7YPV4274LI636YSA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.557595104 (376h58m6.328236439s) [noise=true , address=77.90.31.10:4528 , piece_id=7R3R7QEQIQNYBV7BJF3ZGCMVJFH6OMIXZQTU3KCPSJGQTVT37RRA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.557906210 (376h58m6.328547545s) [noise=true , address=77.90.31.16:4739 , piece_id=A52Q7XZMXUFM5BTN6XYQUAML4HVR55D6IIAY6J7P3OCFTHRIXTVQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.558016280 (376h58m6.328657615s) [noise=true , address=104.36.82.23:4469 , piece_id=N235KELWJET4DXCIBYJGRMR4U5GFVMERXTDM3BM62KR7WBCPJJPA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.558293511 (376h58m6.328934846s) [noise=true , address=104.36.82.41:4783 , piece_id=L73QGLG7NQJR7VC232MVNMZXZWUL75VEISACRGNMH32GT56BFBBQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.558573757 (376h58m6.329215082s) [noise=true , address=104.36.82.2:4757 , piece_id=MGSUR22FTPRP6A7E7DEZNQBFBPOYQBNTGT5ZNEM64FQTCPAAJCTQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.558925379 (376h58m6.329566724s) [noise=true , address=77.90.31.11:4588 , piece_id=HMT3XDYW2QZCZJDKLXV7ISS3JTI7OSFC3HMRKFCZO5FBUH4TP6NA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.559409055 (376h58m6.33005039s) [noise=true , address=79.127.201.214:4305 , piece_id=OL3IFYRAUOVBYLWGO4QXOMDZFGQTCXZAG73F34M7SDSLZ46MNCCA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:32.571736414 (376h58m7.342377739s) [noise=true , address=79.127.201.214:4305 , piece_id=OL3IFYRAUOVBYLWGO4QXOMDZFGQTCXZAG73F34M7SDSLZ46MNCCA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:12HLYBv5M6NE1Njgfz4WdagyLUqPJyc9WjEgr1CfVVEcso7HnCX, address:79.127.201.214:4305): protocol: storage node overloaded, request limit: 2000
Jul 9 02:08:32.707702667 (376h58m7.478344003s) [upload=301581]: Next returned nil segment
Jul 9 02:08:32.707717586 (376h58m7.478358922s) [upload=301581]: Waiting for error group managing segments...
Jul 9 02:08:33.045788856 (376h58m7.816430191s) [noise=true , address=193.149.164.143:4693 , piece_id=ZJKXKNFUYTUKNW25NI5IY5GKBQYNHTAKN6VRLBMYTAMCYKVYXMHQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.477787513 (376h58m8.248428848s) [noise=true , address=45.91.116.7:4041 , piece_id=DI4N5ADKDSKFS6D5HUHXONI63SKBM2BLLRDNTCRD5RWSSKS2W3IQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.492837816 (376h58m8.263479151s) [noise=true , address=77.90.31.17:4780 , piece_id=6XI3ESZNONU2SZJJ3ELUMOMZKSOQY4SBOBRPRBERSTZ2AZRKX57A, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.505858116 (376h58m8.276499451s) [noise=true , address=77.90.31.10:4528 , piece_id=7R3R7QEQIQNYBV7BJF3ZGCMVJFH6OMIXZQTU3KCPSJGQTVT37RRA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.510054294 (376h58m8.280695629s) [noise=true , address=77.90.31.12:4614 , piece_id=5TIDDQKRKPQE3GJ6CMJJZTU5E3LYRTXMMS63G7EAV64Q6AXFWQRA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.511325926 (376h58m8.281967261s) [noise=true , address=77.90.31.11:4588 , piece_id=HMT3XDYW2QZCZJDKLXV7ISS3JTI7OSFC3HMRKFCZO5FBUH4TP6NA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.526917155 (376h58m8.29755849s) [noise=true , address=77.90.31.21:4886 , piece_id=MLVSZ27JZLYNG23IRDFTDMXJRTZ3IMHUOPDGDPOO42GCERLZUQOA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.538345038 (376h58m8.308986373s) [noise=true , address=77.90.31.16:4739 , piece_id=A52Q7XZMXUFM5BTN6XYQUAML4HVR55D6IIAY6J7P3OCFTHRIXTVQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.609285488 (376h58m8.379926823s) [noise=true , address=45.91.116.9:4266 , piece_id=4TFY2OQQ472NIZV4LUREV2BLF75RITUK3OEF43W3KURP2AZYRGCQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:46.984505697 (376h58m21.755147032s) [noise=true , address=104.36.82.20:4441 , piece_id=XSH7ISLZ367ZCC6XM4YN7DDRERMG3MEBFRNH5K6WLXTKIIIPO6SA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:52.339907914 (376h58m27.110549249s) [noise=true , address=104.36.82.41:4783 , piece_id=L73QGLG7NQJR7VC232MVNMZXZWUL75VEISACRGNMH32GT56BFBBQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:09:01.032804568 (376h58m35.803445903s) [noise=true , address=104.36.82.44:4889 , piece_id=SMJJSN3XNTE53GM357QFJMOR2L6EUZP5VG5LH2HLYVG7MFUZJ7AA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:09:08.673338691 (376h58m43.443980026s) [noise=true , address=104.36.82.35:4590 , piece_id=7KGB363U5XFEOK3UJKJXJ57P7T6XBMDUI4UZKQQ3Q4XJ3TJV54XQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:09:30.410919586 (376h59m5.181560921s) [noise=true , address=104.36.82.2:4757 , piece_id=MGSUR22FTPRP6A7E7DEZNQBFBPOYQBNTGT5ZNEM64FQTCPAAJCTQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:104.36.82.2:4757): protocol: expected piece hash; write tcp 188.166.225.170:43178->104.36.82.2:4757: use of closed network connection; write tcp 188.166.225.170:43178->104.36.82.2:4757: use of closed network connection; EOF
Jul 9 02:09:30.410974972 (376h59m5.181616307s) [noise=true , address=104.36.82.40:4944 , piece_id=2DMADEN45IW5WXG6SSK5Y72S2DKZBDBY2NXRZBHWYIICLKWDZFEQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:12KMFsxzp7SfMiFBuMTaWGVyJoRTMw7ZRcfEpPDv1BmbViGF6K9, address:104.36.82.40:4944): protocol: write tcp 188.166.225.170:54250->104.36.82.40:4944: use of closed network connection; write tcp 188.166.225.170:54250->104.36.82.40:4944: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54250->104.36.82.40:4944: use of closed network connection
Jul 9 02:09:30.411014277 (376h59m5.181655612s) [noise=true , address=104.36.82.23:4469 , piece_id=N235KELWJET4DXCIBYJGRMR4U5GFVMERXTDM3BM62KR7WBCPJJPA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:09:30.411035838 (376h59m5.181677174s) [noise=true , address=104.36.82.17:4345 , piece_id=JEG2KMNQ4ZXAYERB65ATCCC6FAYZX5S57GEM7YPV4274LI636YSA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:1ro6Z6UAC6dNRPRWMjdMTTjhg1CvmWPENs7JcqFHxHMpmz5Fnv, address:104.36.82.17:4345): protocol: write tcp 188.166.225.170:41288->104.36.82.17:4345: use of closed network connection; write tcp 188.166.225.170:41288->104.36.82.17:4345: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41288->104.36.82.17:4345: use of closed network connection
Jul 9 02:09:30.411049184 (376h59m5.181690519s) [seg_pos={0 0}, upload=301581]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:09:30.640189311 (376h59m5.410830686s) [upload=301581]: Done uploading segments...
Jul 9 02:09:30.640217886 (376h59m5.410859221s) [upload=301581]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 42.684600869s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.X......................X.................X......X.........X] duration:
30.232575715s
FAIL (2025-07-09 02:10:52 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
_ObjectCommit
Jul 9 02:09:44.378414466 (376h59m19.149055791s) [upload=301592]: Done uploading segments...
Jul 9 02:09:44.378434174 (376h59m19.149075509s) [upload=301592]: Upload finished. err: <nil>
Jul 9 02:10:23.389089694 (376h59m58.159731029s) [upload=301604]: Starting upload
Jul 9 02:10:23.389170209 (376h59m58.159811544s) [upload=301604]: Done starting upload
Jul 9 02:10:23.389354671 (376h59m58.159996006s) [upload=301604]: Uploading segments...
Jul 9 02:10:23.389383707 (376h59m58.160025042s) [upload=301604]: Got next segment. Inline: false
Jul 9 02:10:23.389407963 (376h59m58.160049298s) [upload=301604]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:10:23.389412512 (376h59m58.160053847s) [upload=301604]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:10:23.621494488 (376h59m58.392135823s) [seg_pos={0 0}, upload=301604]: Begin upload segment...
Jul 9 02:10:23.621544643 (376h59m58.392185978s) [seg_pos={0 0}, upload=301604]: Done begin upload segment.
Jul 9 02:10:23.621574050 (376h59m58.392215385s) [noise=true , address=45.91.116.8:4048 , piece_id=SQWCGDZ637P2WH5SN46I5JDBWLGWAOMLZDETZRAJ2QF7AZAIYLNQ, seg_pos={0 0}, upload=301604]: Uploading piece...
Jul 9 02:10:23.622075079 (376h59m58.392716414s) [noise=true , address=77.90.31.10:4517 , piece_id=ATKYYPELMZKPLSCGXBZUHGAUYHS5CSQKRIQ2UPHNNL5DX5FCOB6Q, seg_pos={0 0}, upload=301604]: Uploading piece...
Jul 9 02:10:23.622203704 (376h59m58.392845039s) [noise=true , address=45.91.116.9:4262 , piece_id=KRX6OIIZIKZQSZLSAIDYIBTL4KKZTZSJLR7TTNAPKE3WJY22YE3A, seg_pos={0 0}, upload=301604]: Uploading piece...
Jul 9 02:10:23.622351708 (376h59m58.392993043s) [noise=true , address=104.36.82.2:4729 , piece_id=E4MNXMTRCE5J5SRRZFH4FPBFC2JPWX5DMBTWN4ODD4T6XQ3LETUA, seg_pos={0 0}, upload=301604]: Uploading piece...
Jul 9 02:10:23.622565677 (376h59m58.393207012s) [noise=true , address=104.36.82.6:4873 , piece_id=CLSHXLDPB43ZXG27FG36ISZ3UKDEOQHEJNGN3IBB3PKXBR55GQWQ, seg_pos={0 0}, upload=301604]: Uploading piece...
Jul 9 02:10:23.622596196 (376h59m58.393237531s) [noise=true , address=45.91.116.6:4975 , piece_id=ZB2Z337S4JWTJO5E5TPBJ26EGPNPK2RQQRCL4ODBBQMR4JK2T76A, seg_pos={0 0}, upload=301604]: Uploading piece...
Jul 9 02:10:23.622822469 (376h59m58.393463804s) [noise=true , address=45.91.116.4:4940 , piece_id=OGGGMWZ2DMZHBN3CXCY4WJGUBH7F5BJNTX46W5TOXU3Z2L44IHPA, seg_pos={0 0}, upload=301604]: Uploading piece...
Jul 9 02:10:23.622854680 (376h59m58.393496015s) [noise=true , address=45.91.116.7:4010 , piece_id=CPUXCLT77XILZQLHU36KMG3VO6YLXL4WPOA7BMZKC2E6Q7VGACUQ, seg_pos={0 0}, upload=301604]: Uploading piece...
Jul 9 02:10:23.623053561 (376h59m58.393694896s) [noise=true , address=104.36.82.16:4293 , piece_id=Z5DODKGPEITXICODAJHWMISU63P6IT3JHYZFGCPM6UMDQBX5KRDA, seg_pos={0 0}, upload=301604]: Uploading piece...
Jul 9 02:10:23.623231131 (376h59m58.393872466s) [noise=true , address=45.91.116.11:4119 , piece_id=LUK6QPDR2N7MW7MD3FYXFJYNHF2RB2RZKTREAWIFPHO2F76MP4DQ, seg_pos={0 0}, upload=301604]: Uploading piece...
Jul 9 02:10:23.623494805 (376h59m58.39413614s) [noise=true , address=104.36.82.10:4077 , piece_id=5T3Q3PP2QMQY3XV7OTDGMXSGWIRARNANBRO3JY3B5TDHYDYFGRRQ, seg_pos={0 0}, upload=301604]: Uploading piece...
Jul 9 02:10:23.623759301 (376h59m58.394400626s) [noise=true , address=45.91.116.12:4225 , piece_id=BVIFVKBHMCTYCVOKYMW3ABYGVG5NVPKL2TYCDFBGIBLRASZBUK5Q, seg_pos={0 0}, upload=301604]: Uploading piece...
Jul 9 02:10:23.624555254 (376h59m58.395196589s) [noise=true , address=104.36.82.12:4149 , piece_id=R6AICHJ2UPUIY3SKAXQIWLCG6LCUWCP3MBTRLVXOGI3VFJEIVNNA, seg_pos={0 0}, upload=301604]: Uploading piece...
Jul 9 02:10:23.624804281 (376h59m58.395445626s) [noise=true , address=77.90.31.14:4662 , piece_id=2DTGEE676TKGRVJLTIRMD5M4GWHMIVVYQP55NBOMPYB4ZFQDMV4A, seg_pos={0 0}, upload=301604]: Uploading piece...
Jul 9 02:10:23.625041193 (376h59m58.395682528s) [noise=true , address=104.36.82.7:4909 , piece_id=7L3E72JNHTFEFXW5VXSUCUXYIFEKIVPAC6L6OKQMHXBOJ7EYCRDQ, seg_pos={0 0}, upload=301604]: Uploading piece...
Jul 9 02:10:23.629946998 (376h59m58.400588333s) [noise=true , address=104.36.82.4:4802 , piece_id=R2GG26R2YVB62RQRA2HOU7FWELYTEEACJNBJPO6ZABW3ZNJO45OA, seg_pos={0 0}, upload=301604]: Uploading piece...
Jul 9 02:10:25.317938423 (377h0m0.088579748s) [upload=301604]: Next returned nil segment
Jul 9 02:10:25.317954262 (377h0m0.088595587s) [upload=301604]: Waiting for error group managing segments...
Jul 9 02:10:25.590814734 (377h0m0.361456069s) [noise=true , address=104.36.82.16:4293 , piece_id=Z5DODKGPEITXICODAJHWMISU63P6IT3JHYZFGCPM6UMDQBX5KRDA, seg_pos={0 0}, upload=301604]: Done uploading piece. err: <nil>
Jul 9 02:10:25.675224933 (377h0m0.445866268s) [noise=true , address=45.91.116.6:4975 , piece_id=ZB2Z337S4JWTJO5E5TPBJ26EGPNPK2RQQRCL4ODBBQMR4JK2T76A, seg_pos={0 0}, upload=301604]: Done uploading piece. err: <nil>
Jul 9 02:10:25.685278797 (377h0m0.455920132s) [noise=true , address=45.91.116.11:4119 , piece_id=LUK6QPDR2N7MW7MD3FYXFJYNHF2RB2RZKTREAWIFPHO2F76MP4DQ, seg_pos={0 0}, upload=301604]: Done uploading piece. err: <nil>
Jul 9 02:10:25.700150459 (377h0m0.470791794s) [noise=true , address=45.91.116.8:4048 , piece_id=SQWCGDZ637P2WH5SN46I5JDBWLGWAOMLZDETZRAJ2QF7AZAIYLNQ, seg_pos={0 0}, upload=301604]: Done uploading piece. err: <nil>
Jul 9 02:10:25.703299163 (377h0m0.473940498s) [noise=true , address=45.91.116.7:4010 , piece_id=CPUXCLT77XILZQLHU36KMG3VO6YLXL4WPOA7BMZKC2E6Q7VGACUQ, seg_pos={0 0}, upload=301604]: Done uploading piece. err: <nil>
Jul 9 02:10:25.710212409 (377h0m0.480853744s) [noise=true , address=77.90.31.10:4517 , piece_id=ATKYYPELMZKPLSCGXBZUHGAUYHS5CSQKRIQ2UPHNNL5DX5FCOB6Q, seg_pos={0 0}, upload=301604]: Done uploading piece. err: <nil>
Jul 9 02:10:25.724572863 (377h0m0.495214198s) [noise=true , address=77.90.31.14:4662 , piece_id=2DTGEE676TKGRVJLTIRMD5M4GWHMIVVYQP55NBOMPYB4ZFQDMV4A, seg_pos={0 0}, upload=301604]: Done uploading piece. err: <nil>
Jul 9 02:10:25.775962328 (377h0m0.546603663s) [noise=true , address=45.91.116.12:4225 , piece_id=BVIFVKBHMCTYCVOKYMW3ABYGVG5NVPKL2TYCDFBGIBLRASZBUK5Q, seg_pos={0 0}, upload=301604]: Done uploading piece. err: <nil>
Jul 9 02:10:25.778352431 (377h0m0.548993766s) [noise=true , address=45.91.116.9:4262 , piece_id=KRX6OIIZIKZQSZLSAIDYIBTL4KKZTZSJLR7TTNAPKE3WJY22YE3A, seg_pos={0 0}, upload=301604]: Done uploading piece. err: <nil>
Jul 9 02:10:27.564207033 (377h0m2.334848388s) [noise=true , address=45.91.116.4:4940 , piece_id=OGGGMWZ2DMZHBN3CXCY4WJGUBH7F5BJNTX46W5TOXU3Z2L44IHPA, seg_pos={0 0}, upload=301604]: Done uploading piece. err: <nil>
Jul 9 02:10:45.218238633 (377h0m19.988879968s) [noise=true , address=104.36.82.2:4729 , piece_id=E4MNXMTRCE5J5SRRZFH4FPBFC2JPWX5DMBTWN4ODD4T6XQ3LETUA, seg_pos={0 0}, upload=301604]: Done uploading piece. err: <nil>
Jul 9 02:10:46.448234785 (377h0m21.21887616s) [noise=true , address=104.36.82.10:4077 , piece_id=5T3Q3PP2QMQY3XV7OTDGMXSGWIRARNANBRO3JY3B5TDHYDYFGRRQ, seg_pos={0 0}, upload=301604]: Done uploading piece. err: <nil>
Jul 9 02:10:52.744197825 (377h0m27.51483916s) [noise=true , address=104.36.82.6:4873 , piece_id=CLSHXLDPB43ZXG27FG36ISZ3UKDEOQHEJNGN3IBB3PKXBR55GQWQ, seg_pos={0 0}, upload=301604]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 188.166.225.170:54132->104.36.82.6:4873: use of closed network connection; write tcp 188.166.225.170:54132->104.36.82.6:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54132->104.36.82.6:4873: use of closed network connection
Jul 9 02:10:52.744317835 (377h0m27.51495917s) [noise=true , address=104.36.82.4:4802 , piece_id=R2GG26R2YVB62RQRA2HOU7FWELYTEEACJNBJPO6ZABW3ZNJO45OA, seg_pos={0 0}, upload=301604]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:33706->104.36.82.4:4802: use of closed network connection; write tcp 188.166.225.170:33706->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33706->104.36.82.4:4802: use of closed network connection
Jul 9 02:10:52.744411975 (377h0m27.51505331s) [noise=true , address=104.36.82.12:4149 , piece_id=R6AICHJ2UPUIY3SKAXQIWLCG6LCUWCP3MBTRLVXOGI3VFJEIVNNA, seg_pos={0 0}, upload=301604]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:59790->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:59790->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59790->104.36.82.12:4149: use of closed network connection
Jul 9 02:10:52.744554227 (377h0m27.515195562s) [noise=true , address=104.36.82.7:4909 , piece_id=7L3E72JNHTFEFXW5VXSUCUXYIFEKIVPAC6L6OKQMHXBOJ7EYCRDQ, seg_pos={0 0}, upload=301604]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:51486->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:51486->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51486->104.36.82.7:4909: use of closed network connection
Jul 9 02:10:52.744571260 (377h0m27.515212585s) [seg_pos={0 0}, upload=301604]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:10:52.974969065 (377h0m27.7456104s) [upload=301604]: Done uploading segments...
Jul 9 02:10:52.974995766 (377h0m27.745637091s) [upload=301604]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.232598338s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [X...X..X............X......X..X...X.X.X.X...X...............] duration:
16.025028262s
FAIL (2025-07-09 02:10:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 16.025051487s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
17 out of 60 is failed (threshold=3) [......XX.XX..X......XXX....XX.....XX...X.XX........X.X......] duration:
26.745951179s
FAIL (2025-07-09 02:10:36 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 26.745963202s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..........X......X..........X.............X...............X.] duration:
49.579574438s
FAIL (2025-07-09 02:10:20 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
7177 (376h58m5.878188512s) [upload=301581]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:08:31.554710025 (376h58m6.32535136s) [seg_pos={0 0}, upload=301581]: Begin upload segment...
Jul 9 02:08:31.554766423 (376h58m6.325407758s) [seg_pos={0 0}, upload=301581]: Done begin upload segment.
Jul 9 02:08:31.554787664 (376h58m6.325428989s) [noise=true , address=104.36.82.40:4944 , piece_id=2DMADEN45IW5WXG6SSK5Y72S2DKZBDBY2NXRZBHWYIICLKWDZFEQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.554824724 (376h58m6.325466059s) [noise=true , address=77.90.31.12:4614 , piece_id=5TIDDQKRKPQE3GJ6CMJJZTU5E3LYRTXMMS63G7EAV64Q6AXFWQRA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.555170586 (376h58m6.325811921s) [noise=true , address=45.91.116.9:4266 , piece_id=4TFY2OQQ472NIZV4LUREV2BLF75RITUK3OEF43W3KURP2AZYRGCQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.555483365 (376h58m6.3261247s) [noise=true , address=104.36.82.44:4889 , piece_id=SMJJSN3XNTE53GM357QFJMOR2L6EUZP5VG5LH2HLYVG7MFUZJ7AA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.555790893 (376h58m6.326432218s) [noise=true , address=45.91.116.7:4041 , piece_id=DI4N5ADKDSKFS6D5HUHXONI63SKBM2BLLRDNTCRD5RWSSKS2W3IQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.556074445 (376h58m6.32671578s) [noise=true , address=77.90.31.17:4780 , piece_id=6XI3ESZNONU2SZJJ3ELUMOMZKSOQY4SBOBRPRBERSTZ2AZRKX57A, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.556202771 (376h58m6.326844106s) [noise=true , address=193.149.164.143:4693 , piece_id=ZJKXKNFUYTUKNW25NI5IY5GKBQYNHTAKN6VRLBMYTAMCYKVYXMHQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.556569773 (376h58m6.327211108s) [noise=true , address=104.36.82.20:4441 , piece_id=XSH7ISLZ367ZCC6XM4YN7DDRERMG3MEBFRNH5K6WLXTKIIIPO6SA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.556904713 (376h58m6.327546048s) [noise=true , address=77.90.31.21:4886 , piece_id=MLVSZ27JZLYNG23IRDFTDMXJRTZ3IMHUOPDGDPOO42GCERLZUQOA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.557071302 (376h58m6.327712637s) [noise=true , address=104.36.82.35:4590 , piece_id=7KGB363U5XFEOK3UJKJXJ57P7T6XBMDUI4UZKQQ3Q4XJ3TJV54XQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.557342130 (376h58m6.327983465s) [noise=true , address=104.36.82.17:4345 , piece_id=JEG2KMNQ4ZXAYERB65ATCCC6FAYZX5S57GEM7YPV4274LI636YSA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.557595104 (376h58m6.328236439s) [noise=true , address=77.90.31.10:4528 , piece_id=7R3R7QEQIQNYBV7BJF3ZGCMVJFH6OMIXZQTU3KCPSJGQTVT37RRA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.557906210 (376h58m6.328547545s) [noise=true , address=77.90.31.16:4739 , piece_id=A52Q7XZMXUFM5BTN6XYQUAML4HVR55D6IIAY6J7P3OCFTHRIXTVQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.558016280 (376h58m6.328657615s) [noise=true , address=104.36.82.23:4469 , piece_id=N235KELWJET4DXCIBYJGRMR4U5GFVMERXTDM3BM62KR7WBCPJJPA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.558293511 (376h58m6.328934846s) [noise=true , address=104.36.82.41:4783 , piece_id=L73QGLG7NQJR7VC232MVNMZXZWUL75VEISACRGNMH32GT56BFBBQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.558573757 (376h58m6.329215082s) [noise=true , address=104.36.82.2:4757 , piece_id=MGSUR22FTPRP6A7E7DEZNQBFBPOYQBNTGT5ZNEM64FQTCPAAJCTQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.558925379 (376h58m6.329566724s) [noise=true , address=77.90.31.11:4588 , piece_id=HMT3XDYW2QZCZJDKLXV7ISS3JTI7OSFC3HMRKFCZO5FBUH4TP6NA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.559409055 (376h58m6.33005039s) [noise=true , address=79.127.201.214:4305 , piece_id=OL3IFYRAUOVBYLWGO4QXOMDZFGQTCXZAG73F34M7SDSLZ46MNCCA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:32.571736414 (376h58m7.342377739s) [noise=true , address=79.127.201.214:4305 , piece_id=OL3IFYRAUOVBYLWGO4QXOMDZFGQTCXZAG73F34M7SDSLZ46MNCCA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:12HLYBv5M6NE1Njgfz4WdagyLUqPJyc9WjEgr1CfVVEcso7HnCX, address:79.127.201.214:4305): protocol: storage node overloaded, request limit: 2000
Jul 9 02:08:32.707702667 (376h58m7.478344003s) [upload=301581]: Next returned nil segment
Jul 9 02:08:32.707717586 (376h58m7.478358922s) [upload=301581]: Waiting for error group managing segments...
Jul 9 02:08:33.045788856 (376h58m7.816430191s) [noise=true , address=193.149.164.143:4693 , piece_id=ZJKXKNFUYTUKNW25NI5IY5GKBQYNHTAKN6VRLBMYTAMCYKVYXMHQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.477787513 (376h58m8.248428848s) [noise=true , address=45.91.116.7:4041 , piece_id=DI4N5ADKDSKFS6D5HUHXONI63SKBM2BLLRDNTCRD5RWSSKS2W3IQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.492837816 (376h58m8.263479151s) [noise=true , address=77.90.31.17:4780 , piece_id=6XI3ESZNONU2SZJJ3ELUMOMZKSOQY4SBOBRPRBERSTZ2AZRKX57A, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.505858116 (376h58m8.276499451s) [noise=true , address=77.90.31.10:4528 , piece_id=7R3R7QEQIQNYBV7BJF3ZGCMVJFH6OMIXZQTU3KCPSJGQTVT37RRA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.510054294 (376h58m8.280695629s) [noise=true , address=77.90.31.12:4614 , piece_id=5TIDDQKRKPQE3GJ6CMJJZTU5E3LYRTXMMS63G7EAV64Q6AXFWQRA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.511325926 (376h58m8.281967261s) [noise=true , address=77.90.31.11:4588 , piece_id=HMT3XDYW2QZCZJDKLXV7ISS3JTI7OSFC3HMRKFCZO5FBUH4TP6NA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.526917155 (376h58m8.29755849s) [noise=true , address=77.90.31.21:4886 , piece_id=MLVSZ27JZLYNG23IRDFTDMXJRTZ3IMHUOPDGDPOO42GCERLZUQOA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.538345038 (376h58m8.308986373s) [noise=true , address=77.90.31.16:4739 , piece_id=A52Q7XZMXUFM5BTN6XYQUAML4HVR55D6IIAY6J7P3OCFTHRIXTVQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.609285488 (376h58m8.379926823s) [noise=true , address=45.91.116.9:4266 , piece_id=4TFY2OQQ472NIZV4LUREV2BLF75RITUK3OEF43W3KURP2AZYRGCQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:46.984505697 (376h58m21.755147032s) [noise=true , address=104.36.82.20:4441 , piece_id=XSH7ISLZ367ZCC6XM4YN7DDRERMG3MEBFRNH5K6WLXTKIIIPO6SA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:52.339907914 (376h58m27.110549249s) [noise=true , address=104.36.82.41:4783 , piece_id=L73QGLG7NQJR7VC232MVNMZXZWUL75VEISACRGNMH32GT56BFBBQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:09:01.032804568 (376h58m35.803445903s) [noise=true , address=104.36.82.44:4889 , piece_id=SMJJSN3XNTE53GM357QFJMOR2L6EUZP5VG5LH2HLYVG7MFUZJ7AA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:09:08.673338691 (376h58m43.443980026s) [noise=true , address=104.36.82.35:4590 , piece_id=7KGB363U5XFEOK3UJKJXJ57P7T6XBMDUI4UZKQQ3Q4XJ3TJV54XQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:09:30.410919586 (376h59m5.181560921s) [noise=true , address=104.36.82.2:4757 , piece_id=MGSUR22FTPRP6A7E7DEZNQBFBPOYQBNTGT5ZNEM64FQTCPAAJCTQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:104.36.82.2:4757): protocol: expected piece hash; write tcp 188.166.225.170:43178->104.36.82.2:4757: use of closed network connection; write tcp 188.166.225.170:43178->104.36.82.2:4757: use of closed network connection; EOF
Jul 9 02:09:30.410974972 (376h59m5.181616307s) [noise=true , address=104.36.82.40:4944 , piece_id=2DMADEN45IW5WXG6SSK5Y72S2DKZBDBY2NXRZBHWYIICLKWDZFEQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:12KMFsxzp7SfMiFBuMTaWGVyJoRTMw7ZRcfEpPDv1BmbViGF6K9, address:104.36.82.40:4944): protocol: write tcp 188.166.225.170:54250->104.36.82.40:4944: use of closed network connection; write tcp 188.166.225.170:54250->104.36.82.40:4944: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54250->104.36.82.40:4944: use of closed network connection
Jul 9 02:09:30.411014277 (376h59m5.181655612s) [noise=true , address=104.36.82.23:4469 , piece_id=N235KELWJET4DXCIBYJGRMR4U5GFVMERXTDM3BM62KR7WBCPJJPA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:09:30.411035838 (376h59m5.181677174s) [noise=true , address=104.36.82.17:4345 , piece_id=JEG2KMNQ4ZXAYERB65ATCCC6FAYZX5S57GEM7YPV4274LI636YSA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:1ro6Z6UAC6dNRPRWMjdMTTjhg1CvmWPENs7JcqFHxHMpmz5Fnv, address:104.36.82.17:4345): protocol: write tcp 188.166.225.170:41288->104.36.82.17:4345: use of closed network connection; write tcp 188.166.225.170:41288->104.36.82.17:4345: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41288->104.36.82.17:4345: use of closed network connection
Jul 9 02:09:30.411049184 (376h59m5.181690519s) [seg_pos={0 0}, upload=301581]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:09:30.640189311 (376h59m5.410830686s) [upload=301581]: Done uploading segments...
Jul 9 02:09:30.640217886 (376h59m5.410859221s) [upload=301581]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 49.579591672s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [X......XX.XX..X......XXX....XX.....XX...X.XX........X.X.....] duration:
48.708397736s
FAIL (2025-07-09 02:10:09 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 48.708418535s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X......................X.................X......X.........] duration:
17.1410682s
FAIL (2025-07-09 02:09:46 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
nceled; EOF
Jul 9 02:00:49.060314467 (376h50m23.830955812s) [seg_pos={0 0}, upload=301481]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Jul 9 02:00:49.060367628 (376h50m23.831008963s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:00:49.060371145 (376h50m23.83101248s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:00:49.357373945 (376h50m24.12801528s) [upload=301481]: Done uploading segments...
Jul 9 02:00:49.357395647 (376h50m24.128036982s) [upload=301481]: Upload finished. err: <nil>
Jul 9 02:02:01.314289870 (376h51m36.084931205s) [upload=301495]: Starting upload
Jul 9 02:02:01.314394990 (376h51m36.085036325s) [upload=301495]: Done starting upload
Jul 9 02:02:01.314484112 (376h51m36.085125457s) [upload=301495]: Uploading segments...
Jul 9 02:02:01.314699112 (376h51m36.085340457s) [upload=301495]: Got next segment. Inline: false
Jul 9 02:02:01.314747194 (376h51m36.085388529s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:01.314753927 (376h51m36.085395272s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:01.549200488 (376h51m36.319841873s) [seg_pos={0 0}, upload=301495]: Begin upload segment...
Jul 9 02:02:01.549266264 (376h51m36.319907609s) [seg_pos={0 0}, upload=301495]: Done begin upload segment.
Jul 9 02:02:01.549310128 (376h51m36.319951473s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549336468 (376h51m36.319977823s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549964509 (376h51m36.320605844s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550253653 (376h51m36.320894988s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550518870 (376h51m36.321160225s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550556472 (376h51m36.321197807s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550837249 (376h51m36.321478584s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551078060 (376h51m36.321719395s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551313831 (376h51m36.321955166s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551369758 (376h51m36.322011093s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551601652 (376h51m36.322242987s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551803969 (376h51m36.322445304s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552051763 (376h51m36.322693098s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552343200 (376h51m36.322984545s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552474742 (376h51m36.323116077s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552718097 (376h51m36.323359433s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:03.251350348 (376h51m38.021991683s) [upload=301495]: Next returned nil segment
Jul 9 02:02:03.251367120 (376h51m38.022008455s) [upload=301495]: Waiting for error group managing segments...
Jul 9 02:02:03.536295419 (376h51m38.306936775s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.585972469 (376h51m38.356613794s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.612355463 (376h51m38.382996798s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.616140875 (376h51m38.38678221s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.625063725 (376h51m38.395705061s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.627327436 (376h51m38.397968771s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.680079459 (376h51m38.450720804s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.682275680 (376h51m38.452917015s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.693418348 (376h51m38.464059684s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.714112589 (376h51m38.484753924s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:24.743005877 (376h51m59.513647202s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:28.578275204 (376h52m3.348916539s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:30.664723094 (376h52m5.435364439s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:02:30.665014852 (376h52m5.435656188s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection
Jul 9 02:02:30.665113180 (376h52m5.435754525s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection
Jul 9 02:02:30.665314215 (376h52m5.43595555s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:50202->104.36.82.15:4258: use of closed network connection; context deadline exceeded
Jul 9 02:02:30.665353560 (376h52m5.435994895s) [seg_pos={0 0}, upload=301495]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:02:30.900452118 (376h52m5.671093453s) [upload=301495]: Done uploading segments...
Jul 9 02:02:30.900486934 (376h52m5.67112827s) [upload=301495]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 17.14109914s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.X...X..X............X......X..X...X.X.X.X...X..............] duration:
5.940192755s
FAIL (2025-07-09 02:09:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.940213134s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...........X......X..........X.............X...............X] duration:
1m0.230845939s
FAIL (2025-07-09 02:09:30 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
7177 (376h58m5.878188512s) [upload=301581]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:08:31.554710025 (376h58m6.32535136s) [seg_pos={0 0}, upload=301581]: Begin upload segment...
Jul 9 02:08:31.554766423 (376h58m6.325407758s) [seg_pos={0 0}, upload=301581]: Done begin upload segment.
Jul 9 02:08:31.554787664 (376h58m6.325428989s) [noise=true , address=104.36.82.40:4944 , piece_id=2DMADEN45IW5WXG6SSK5Y72S2DKZBDBY2NXRZBHWYIICLKWDZFEQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.554824724 (376h58m6.325466059s) [noise=true , address=77.90.31.12:4614 , piece_id=5TIDDQKRKPQE3GJ6CMJJZTU5E3LYRTXMMS63G7EAV64Q6AXFWQRA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.555170586 (376h58m6.325811921s) [noise=true , address=45.91.116.9:4266 , piece_id=4TFY2OQQ472NIZV4LUREV2BLF75RITUK3OEF43W3KURP2AZYRGCQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.555483365 (376h58m6.3261247s) [noise=true , address=104.36.82.44:4889 , piece_id=SMJJSN3XNTE53GM357QFJMOR2L6EUZP5VG5LH2HLYVG7MFUZJ7AA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.555790893 (376h58m6.326432218s) [noise=true , address=45.91.116.7:4041 , piece_id=DI4N5ADKDSKFS6D5HUHXONI63SKBM2BLLRDNTCRD5RWSSKS2W3IQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.556074445 (376h58m6.32671578s) [noise=true , address=77.90.31.17:4780 , piece_id=6XI3ESZNONU2SZJJ3ELUMOMZKSOQY4SBOBRPRBERSTZ2AZRKX57A, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.556202771 (376h58m6.326844106s) [noise=true , address=193.149.164.143:4693 , piece_id=ZJKXKNFUYTUKNW25NI5IY5GKBQYNHTAKN6VRLBMYTAMCYKVYXMHQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.556569773 (376h58m6.327211108s) [noise=true , address=104.36.82.20:4441 , piece_id=XSH7ISLZ367ZCC6XM4YN7DDRERMG3MEBFRNH5K6WLXTKIIIPO6SA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.556904713 (376h58m6.327546048s) [noise=true , address=77.90.31.21:4886 , piece_id=MLVSZ27JZLYNG23IRDFTDMXJRTZ3IMHUOPDGDPOO42GCERLZUQOA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.557071302 (376h58m6.327712637s) [noise=true , address=104.36.82.35:4590 , piece_id=7KGB363U5XFEOK3UJKJXJ57P7T6XBMDUI4UZKQQ3Q4XJ3TJV54XQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.557342130 (376h58m6.327983465s) [noise=true , address=104.36.82.17:4345 , piece_id=JEG2KMNQ4ZXAYERB65ATCCC6FAYZX5S57GEM7YPV4274LI636YSA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.557595104 (376h58m6.328236439s) [noise=true , address=77.90.31.10:4528 , piece_id=7R3R7QEQIQNYBV7BJF3ZGCMVJFH6OMIXZQTU3KCPSJGQTVT37RRA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.557906210 (376h58m6.328547545s) [noise=true , address=77.90.31.16:4739 , piece_id=A52Q7XZMXUFM5BTN6XYQUAML4HVR55D6IIAY6J7P3OCFTHRIXTVQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.558016280 (376h58m6.328657615s) [noise=true , address=104.36.82.23:4469 , piece_id=N235KELWJET4DXCIBYJGRMR4U5GFVMERXTDM3BM62KR7WBCPJJPA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.558293511 (376h58m6.328934846s) [noise=true , address=104.36.82.41:4783 , piece_id=L73QGLG7NQJR7VC232MVNMZXZWUL75VEISACRGNMH32GT56BFBBQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.558573757 (376h58m6.329215082s) [noise=true , address=104.36.82.2:4757 , piece_id=MGSUR22FTPRP6A7E7DEZNQBFBPOYQBNTGT5ZNEM64FQTCPAAJCTQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.558925379 (376h58m6.329566724s) [noise=true , address=77.90.31.11:4588 , piece_id=HMT3XDYW2QZCZJDKLXV7ISS3JTI7OSFC3HMRKFCZO5FBUH4TP6NA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.559409055 (376h58m6.33005039s) [noise=true , address=79.127.201.214:4305 , piece_id=OL3IFYRAUOVBYLWGO4QXOMDZFGQTCXZAG73F34M7SDSLZ46MNCCA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:32.571736414 (376h58m7.342377739s) [noise=true , address=79.127.201.214:4305 , piece_id=OL3IFYRAUOVBYLWGO4QXOMDZFGQTCXZAG73F34M7SDSLZ46MNCCA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:12HLYBv5M6NE1Njgfz4WdagyLUqPJyc9WjEgr1CfVVEcso7HnCX, address:79.127.201.214:4305): protocol: storage node overloaded, request limit: 2000
Jul 9 02:08:32.707702667 (376h58m7.478344003s) [upload=301581]: Next returned nil segment
Jul 9 02:08:32.707717586 (376h58m7.478358922s) [upload=301581]: Waiting for error group managing segments...
Jul 9 02:08:33.045788856 (376h58m7.816430191s) [noise=true , address=193.149.164.143:4693 , piece_id=ZJKXKNFUYTUKNW25NI5IY5GKBQYNHTAKN6VRLBMYTAMCYKVYXMHQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.477787513 (376h58m8.248428848s) [noise=true , address=45.91.116.7:4041 , piece_id=DI4N5ADKDSKFS6D5HUHXONI63SKBM2BLLRDNTCRD5RWSSKS2W3IQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.492837816 (376h58m8.263479151s) [noise=true , address=77.90.31.17:4780 , piece_id=6XI3ESZNONU2SZJJ3ELUMOMZKSOQY4SBOBRPRBERSTZ2AZRKX57A, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.505858116 (376h58m8.276499451s) [noise=true , address=77.90.31.10:4528 , piece_id=7R3R7QEQIQNYBV7BJF3ZGCMVJFH6OMIXZQTU3KCPSJGQTVT37RRA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.510054294 (376h58m8.280695629s) [noise=true , address=77.90.31.12:4614 , piece_id=5TIDDQKRKPQE3GJ6CMJJZTU5E3LYRTXMMS63G7EAV64Q6AXFWQRA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.511325926 (376h58m8.281967261s) [noise=true , address=77.90.31.11:4588 , piece_id=HMT3XDYW2QZCZJDKLXV7ISS3JTI7OSFC3HMRKFCZO5FBUH4TP6NA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.526917155 (376h58m8.29755849s) [noise=true , address=77.90.31.21:4886 , piece_id=MLVSZ27JZLYNG23IRDFTDMXJRTZ3IMHUOPDGDPOO42GCERLZUQOA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.538345038 (376h58m8.308986373s) [noise=true , address=77.90.31.16:4739 , piece_id=A52Q7XZMXUFM5BTN6XYQUAML4HVR55D6IIAY6J7P3OCFTHRIXTVQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.609285488 (376h58m8.379926823s) [noise=true , address=45.91.116.9:4266 , piece_id=4TFY2OQQ472NIZV4LUREV2BLF75RITUK3OEF43W3KURP2AZYRGCQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:46.984505697 (376h58m21.755147032s) [noise=true , address=104.36.82.20:4441 , piece_id=XSH7ISLZ367ZCC6XM4YN7DDRERMG3MEBFRNH5K6WLXTKIIIPO6SA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:52.339907914 (376h58m27.110549249s) [noise=true , address=104.36.82.41:4783 , piece_id=L73QGLG7NQJR7VC232MVNMZXZWUL75VEISACRGNMH32GT56BFBBQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:09:01.032804568 (376h58m35.803445903s) [noise=true , address=104.36.82.44:4889 , piece_id=SMJJSN3XNTE53GM357QFJMOR2L6EUZP5VG5LH2HLYVG7MFUZJ7AA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:09:08.673338691 (376h58m43.443980026s) [noise=true , address=104.36.82.35:4590 , piece_id=7KGB363U5XFEOK3UJKJXJ57P7T6XBMDUI4UZKQQ3Q4XJ3TJV54XQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:09:30.410919586 (376h59m5.181560921s) [noise=true , address=104.36.82.2:4757 , piece_id=MGSUR22FTPRP6A7E7DEZNQBFBPOYQBNTGT5ZNEM64FQTCPAAJCTQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:104.36.82.2:4757): protocol: expected piece hash; write tcp 188.166.225.170:43178->104.36.82.2:4757: use of closed network connection; write tcp 188.166.225.170:43178->104.36.82.2:4757: use of closed network connection; EOF
Jul 9 02:09:30.410974972 (376h59m5.181616307s) [noise=true , address=104.36.82.40:4944 , piece_id=2DMADEN45IW5WXG6SSK5Y72S2DKZBDBY2NXRZBHWYIICLKWDZFEQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:12KMFsxzp7SfMiFBuMTaWGVyJoRTMw7ZRcfEpPDv1BmbViGF6K9, address:104.36.82.40:4944): protocol: write tcp 188.166.225.170:54250->104.36.82.40:4944: use of closed network connection; write tcp 188.166.225.170:54250->104.36.82.40:4944: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54250->104.36.82.40:4944: use of closed network connection
Jul 9 02:09:30.411014277 (376h59m5.181655612s) [noise=true , address=104.36.82.23:4469 , piece_id=N235KELWJET4DXCIBYJGRMR4U5GFVMERXTDM3BM62KR7WBCPJJPA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:09:30.411035838 (376h59m5.181677174s) [noise=true , address=104.36.82.17:4345 , piece_id=JEG2KMNQ4ZXAYERB65ATCCC6FAYZX5S57GEM7YPV4274LI636YSA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:1ro6Z6UAC6dNRPRWMjdMTTjhg1CvmWPENs7JcqFHxHMpmz5Fnv, address:104.36.82.17:4345): protocol: write tcp 188.166.225.170:41288->104.36.82.17:4345: use of closed network connection; write tcp 188.166.225.170:41288->104.36.82.17:4345: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41288->104.36.82.17:4345: use of closed network connection
Jul 9 02:09:30.411049184 (376h59m5.181690519s) [seg_pos={0 0}, upload=301581]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:09:30.640189311 (376h59m5.410830686s) [upload=301581]: Done uploading segments...
Jul 9 02:09:30.640217886 (376h59m5.410859221s) [upload=301581]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.230858192s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [.X......XX.XX..X......XXX....XX.....XX...X.XX........X.X....] duration:
29.364243386s
FAIL (2025-07-09 02:09:20 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 29.364258896s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [..X...X..X............X......X..X...X.X.X.X...X.............] duration:
13.976334888s
FAIL (2025-07-09 02:08:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 13.976358723s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [..X......XX.XX..X......XXX....XX.....XX...X.XX........X.X...] duration:
42.884576715s
FAIL (2025-07-09 02:08:37 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 42.884619578s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X......................X.................X......X........] duration:
5.690140913s
FAIL (2025-07-09 02:08:24 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
nceled; EOF
Jul 9 02:00:49.060314467 (376h50m23.830955812s) [seg_pos={0 0}, upload=301481]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Jul 9 02:00:49.060367628 (376h50m23.831008963s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:00:49.060371145 (376h50m23.83101248s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:00:49.357373945 (376h50m24.12801528s) [upload=301481]: Done uploading segments...
Jul 9 02:00:49.357395647 (376h50m24.128036982s) [upload=301481]: Upload finished. err: <nil>
Jul 9 02:02:01.314289870 (376h51m36.084931205s) [upload=301495]: Starting upload
Jul 9 02:02:01.314394990 (376h51m36.085036325s) [upload=301495]: Done starting upload
Jul 9 02:02:01.314484112 (376h51m36.085125457s) [upload=301495]: Uploading segments...
Jul 9 02:02:01.314699112 (376h51m36.085340457s) [upload=301495]: Got next segment. Inline: false
Jul 9 02:02:01.314747194 (376h51m36.085388529s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:01.314753927 (376h51m36.085395272s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:01.549200488 (376h51m36.319841873s) [seg_pos={0 0}, upload=301495]: Begin upload segment...
Jul 9 02:02:01.549266264 (376h51m36.319907609s) [seg_pos={0 0}, upload=301495]: Done begin upload segment.
Jul 9 02:02:01.549310128 (376h51m36.319951473s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549336468 (376h51m36.319977823s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549964509 (376h51m36.320605844s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550253653 (376h51m36.320894988s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550518870 (376h51m36.321160225s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550556472 (376h51m36.321197807s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550837249 (376h51m36.321478584s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551078060 (376h51m36.321719395s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551313831 (376h51m36.321955166s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551369758 (376h51m36.322011093s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551601652 (376h51m36.322242987s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551803969 (376h51m36.322445304s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552051763 (376h51m36.322693098s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552343200 (376h51m36.322984545s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552474742 (376h51m36.323116077s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552718097 (376h51m36.323359433s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:03.251350348 (376h51m38.021991683s) [upload=301495]: Next returned nil segment
Jul 9 02:02:03.251367120 (376h51m38.022008455s) [upload=301495]: Waiting for error group managing segments...
Jul 9 02:02:03.536295419 (376h51m38.306936775s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.585972469 (376h51m38.356613794s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.612355463 (376h51m38.382996798s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.616140875 (376h51m38.38678221s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.625063725 (376h51m38.395705061s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.627327436 (376h51m38.397968771s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.680079459 (376h51m38.450720804s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.682275680 (376h51m38.452917015s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.693418348 (376h51m38.464059684s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.714112589 (376h51m38.484753924s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:24.743005877 (376h51m59.513647202s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:28.578275204 (376h52m3.348916539s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:30.664723094 (376h52m5.435364439s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:02:30.665014852 (376h52m5.435656188s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection
Jul 9 02:02:30.665113180 (376h52m5.435754525s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection
Jul 9 02:02:30.665314215 (376h52m5.43595555s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:50202->104.36.82.15:4258: use of closed network connection; context deadline exceeded
Jul 9 02:02:30.665353560 (376h52m5.435994895s) [seg_pos={0 0}, upload=301495]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:02:30.900452118 (376h52m5.671093453s) [upload=301495]: Done uploading segments...
Jul 9 02:02:30.900486934 (376h52m5.67112827s) [upload=301495]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.690167081s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [X..X...X..X............X......X..X...X.X.X.X...X............] duration:
9.528807289s
FAIL (2025-07-09 02:08:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.528836173s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [............X......X..........X.............X...............] duration:
34.405783824s
FAIL (2025-07-09 02:07:53 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
hRequestItem_SegmentBegin
Jul 9 01:56:00.629444016 (376h45m35.400085361s) [seg_pos={0 0}, upload=301417]: Begin upload segment...
Jul 9 01:56:00.629512196 (376h45m35.400153531s) [seg_pos={0 0}, upload=301417]: Done begin upload segment.
Jul 9 01:56:00.629572552 (376h45m35.400213887s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629579284 (376h45m35.400220619s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629690497 (376h45m35.400331832s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630082818 (376h45m35.400724153s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630158242 (376h45m35.400799577s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630428680 (376h45m35.401070025s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630538129 (376h45m35.401179464s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630862820 (376h45m35.401504165s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631210084 (376h45m35.401851419s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631555495 (376h45m35.40219684s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631989165 (376h45m35.4026305s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632258450 (376h45m35.402899785s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632521984 (376h45m35.403163309s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632835304 (376h45m35.403476649s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.633120529 (376h45m35.403761864s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636119667 (376h45m35.406761002s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636833744 (376h45m35.407475079s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.637112587 (376h45m35.407753922s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:02.137197163 (376h45m36.907838498s) [upload=301417]: Next returned nil segment
Jul 9 01:56:02.137214166 (376h45m36.907855501s) [upload=301417]: Waiting for error group managing segments...
Jul 9 01:56:02.521317983 (376h45m37.291959318s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.567664820 (376h45m37.338306156s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.568760968 (376h45m37.339402303s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.573315612 (376h45m37.343956947s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.577761728 (376h45m37.348403053s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.579676210 (376h45m37.350317545s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.595175322 (376h45m37.365816657s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.601273909 (376h45m37.371915244s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:07.782538378 (376h45m42.553179714s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:22.662088184 (376h45m57.432729519s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:24.053592013 (376h45m58.824233348s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:44.900784468 (376h46m19.671425793s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:47.726372911 (376h46m22.497014236s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047624046 (376h46m30.818265381s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047644345 (376h46m30.81828568s) [seg_pos={0 0}, upload=301417]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:56:56.047718136 (376h46m30.818359471s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047756840 (376h46m30.818398175s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047948587 (376h46m30.818589922s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3): protocol: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection
Jul 9 01:56:56.047987552 (376h46m30.818628887s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125NsQhq7kRmTjKS8nuc51PZddApFEi5C38FzC5pEX2QrWYKzSW): protocol: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection
Jul 9 01:56:56.047996138 (376h46m30.818637463s) [seg_pos={0 0}, upload=301417]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:56:56.048046284 (376h46m30.818687619s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:56:56.048050091 (376h46m30.818691416s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:56:56.339173192 (376h46m31.109814527s) [upload=301417]: Done uploading segments...
Jul 9 01:56:56.339190416 (376h46m31.109831751s) [upload=301417]: Upload finished. err: <nil>
Duration: 34.405806959s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [...X......XX.XX..X......XXX....XX.....XX...X.XX........X.X..] duration:
36.765325027s
FAIL (2025-07-09 02:07:31 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 36.765345346s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [....X......................X.................X......X.......] duration:
5.702014777s
FAIL (2025-07-09 02:07:22 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
nceled; EOF
Jul 9 02:00:49.060314467 (376h50m23.830955812s) [seg_pos={0 0}, upload=301481]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Jul 9 02:00:49.060367628 (376h50m23.831008963s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:00:49.060371145 (376h50m23.83101248s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:00:49.357373945 (376h50m24.12801528s) [upload=301481]: Done uploading segments...
Jul 9 02:00:49.357395647 (376h50m24.128036982s) [upload=301481]: Upload finished. err: <nil>
Jul 9 02:02:01.314289870 (376h51m36.084931205s) [upload=301495]: Starting upload
Jul 9 02:02:01.314394990 (376h51m36.085036325s) [upload=301495]: Done starting upload
Jul 9 02:02:01.314484112 (376h51m36.085125457s) [upload=301495]: Uploading segments...
Jul 9 02:02:01.314699112 (376h51m36.085340457s) [upload=301495]: Got next segment. Inline: false
Jul 9 02:02:01.314747194 (376h51m36.085388529s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:01.314753927 (376h51m36.085395272s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:01.549200488 (376h51m36.319841873s) [seg_pos={0 0}, upload=301495]: Begin upload segment...
Jul 9 02:02:01.549266264 (376h51m36.319907609s) [seg_pos={0 0}, upload=301495]: Done begin upload segment.
Jul 9 02:02:01.549310128 (376h51m36.319951473s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549336468 (376h51m36.319977823s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549964509 (376h51m36.320605844s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550253653 (376h51m36.320894988s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550518870 (376h51m36.321160225s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550556472 (376h51m36.321197807s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550837249 (376h51m36.321478584s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551078060 (376h51m36.321719395s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551313831 (376h51m36.321955166s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551369758 (376h51m36.322011093s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551601652 (376h51m36.322242987s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551803969 (376h51m36.322445304s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552051763 (376h51m36.322693098s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552343200 (376h51m36.322984545s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552474742 (376h51m36.323116077s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552718097 (376h51m36.323359433s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:03.251350348 (376h51m38.021991683s) [upload=301495]: Next returned nil segment
Jul 9 02:02:03.251367120 (376h51m38.022008455s) [upload=301495]: Waiting for error group managing segments...
Jul 9 02:02:03.536295419 (376h51m38.306936775s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.585972469 (376h51m38.356613794s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.612355463 (376h51m38.382996798s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.616140875 (376h51m38.38678221s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.625063725 (376h51m38.395705061s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.627327436 (376h51m38.397968771s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.680079459 (376h51m38.450720804s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.682275680 (376h51m38.452917015s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.693418348 (376h51m38.464059684s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.714112589 (376h51m38.484753924s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:24.743005877 (376h51m59.513647202s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:28.578275204 (376h52m3.348916539s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:30.664723094 (376h52m5.435364439s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:02:30.665014852 (376h52m5.435656188s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection
Jul 9 02:02:30.665113180 (376h52m5.435754525s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection
Jul 9 02:02:30.665314215 (376h52m5.43595555s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:50202->104.36.82.15:4258: use of closed network connection; context deadline exceeded
Jul 9 02:02:30.665353560 (376h52m5.435994895s) [seg_pos={0 0}, upload=301495]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:02:30.900452118 (376h52m5.671093453s) [upload=301495]: Done uploading segments...
Jul 9 02:02:30.900486934 (376h52m5.67112827s) [upload=301495]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.702040697s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [.X..X...X..X............X......X..X...X.X.X.X...X...........] duration:
8.869038877s
FAIL (2025-07-09 02:07:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.869058925s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.............X......X..........X.............X..............] duration:
26.21498432s
FAIL (2025-07-09 02:07:05 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
hRequestItem_SegmentBegin
Jul 9 01:56:00.629444016 (376h45m35.400085361s) [seg_pos={0 0}, upload=301417]: Begin upload segment...
Jul 9 01:56:00.629512196 (376h45m35.400153531s) [seg_pos={0 0}, upload=301417]: Done begin upload segment.
Jul 9 01:56:00.629572552 (376h45m35.400213887s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629579284 (376h45m35.400220619s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629690497 (376h45m35.400331832s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630082818 (376h45m35.400724153s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630158242 (376h45m35.400799577s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630428680 (376h45m35.401070025s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630538129 (376h45m35.401179464s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630862820 (376h45m35.401504165s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631210084 (376h45m35.401851419s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631555495 (376h45m35.40219684s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631989165 (376h45m35.4026305s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632258450 (376h45m35.402899785s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632521984 (376h45m35.403163309s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632835304 (376h45m35.403476649s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.633120529 (376h45m35.403761864s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636119667 (376h45m35.406761002s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636833744 (376h45m35.407475079s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.637112587 (376h45m35.407753922s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:02.137197163 (376h45m36.907838498s) [upload=301417]: Next returned nil segment
Jul 9 01:56:02.137214166 (376h45m36.907855501s) [upload=301417]: Waiting for error group managing segments...
Jul 9 01:56:02.521317983 (376h45m37.291959318s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.567664820 (376h45m37.338306156s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.568760968 (376h45m37.339402303s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.573315612 (376h45m37.343956947s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.577761728 (376h45m37.348403053s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.579676210 (376h45m37.350317545s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.595175322 (376h45m37.365816657s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.601273909 (376h45m37.371915244s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:07.782538378 (376h45m42.553179714s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:22.662088184 (376h45m57.432729519s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:24.053592013 (376h45m58.824233348s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:44.900784468 (376h46m19.671425793s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:47.726372911 (376h46m22.497014236s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047624046 (376h46m30.818265381s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047644345 (376h46m30.81828568s) [seg_pos={0 0}, upload=301417]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:56:56.047718136 (376h46m30.818359471s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047756840 (376h46m30.818398175s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047948587 (376h46m30.818589922s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3): protocol: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection
Jul 9 01:56:56.047987552 (376h46m30.818628887s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125NsQhq7kRmTjKS8nuc51PZddApFEi5C38FzC5pEX2QrWYKzSW): protocol: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection
Jul 9 01:56:56.047996138 (376h46m30.818637463s) [seg_pos={0 0}, upload=301417]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:56:56.048046284 (376h46m30.818687619s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:56:56.048050091 (376h46m30.818691416s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:56:56.339173192 (376h46m31.109814527s) [upload=301417]: Done uploading segments...
Jul 9 01:56:56.339190416 (376h46m31.109831751s) [upload=301417]: Upload finished. err: <nil>
Duration: 26.215001013s
core
uplink
us-select-1
us1-placement-20
|