Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...X........X..X..X....XX..XXX.XX..X.XX..X........XX..XX...X] duration: 30.00166864s FAIL (2025-10-08 13:02:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ct 8 13:01:57.986076116 (359h30m43.448168599s) [upload=316311]: Starting upload Oct 8 13:01:57.986153663 (359h30m43.448246146s) [upload=316311]: Done starting upload Oct 8 13:01:57.986259396 (359h30m43.448351879s) [upload=316311]: Uploading segments... Oct 8 13:01:57.986408610 (359h30m43.448501083s) [upload=316311]: Got next segment. Inline: false Oct 8 13:01:57.986445650 (359h30m43.448538123s) [upload=316311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:01:57.986451090 (359h30m43.448543563s) [upload=316311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:01:58.252449976 (359h30m43.714542469s) [seg_pos={0 0}, upload=316311]: Begin upload segment... Oct 8 13:01:58.252567730 (359h30m43.714660213s) [seg_pos={0 0}, upload=316311]: Done begin upload segment. Oct 8 13:01:58.252603878 (359h30m43.714696361s) [noise=true , address=45.91.116.12:4225 , piece_id=6WTRWQ625W6WUV2DS27PNLPEWPPSR52H3YFOT6SCMG6TXP6W3J2A, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.253640212 (359h30m43.715732705s) [noise=true , address=45.91.116.6:4975 , piece_id=P5JV6USSWDB335IRLXXCWMJEHYBUDN6KKCYSTJRWDLUEJEOY7RRA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.254216099 (359h30m43.716308582s) [noise=true , address=145.79.227.18:4005 , piece_id=6IB6TFKCAM7BEUQAMJESVPYV4HNGFZXV7R5YBT5NOM3IJKRBIRPQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.254793638 (359h30m43.716886111s) [noise=true , address=45.91.116.6:4973 , piece_id=GW7S6KTE5DLEUMUAEJE27VOFZY5YSTLOPJRCMKRZJCO3WCTQIJEQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.255413679 (359h30m43.717506152s) [noise=true , address=45.91.116.7:4009 , piece_id=HUHKWUSQB5LGFWPQZHHS6NTISCESL4WCZ5E5XJCDHK55FBWB6XXA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.255989366 (359h30m43.718081849s) [noise=true , address=77.90.31.10:4519 , piece_id=GVBAGJMLRINBGUGLR7K5U7U5UDAUO2TDHVQU3YOGH4LUMOZUJP4A, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.256475141 (359h30m43.718567614s) [noise=true , address=45.91.116.8:4045 , piece_id=RH3GWGTRTNKP2BZWOPUQSB75VRK63CPWKYJBGQ5PUYACKF46BGRQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.256916521 (359h30m43.719009005s) [noise=true , address=77.90.31.17:4771 , piece_id=3PXQBNPKNSPGWX457DDSCF7GTKENAKTPLOPHE22LLNFBT5SAZMRQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.257649487 (359h30m43.71974197s) [noise=true , address=45.91.116.14:4191 , piece_id=OB5O5DHWAVFLR64FIY2JMETOA6YBVHQZ7X623XG37FISJBLWQKKQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.258326596 (359h30m43.720419069s) [noise=true , address=145.79.227.13:4804 , piece_id=5CIOSYYYG2YEYYDM44JAGWEONLRJU2VJOCR6SA3TIIRB4DCBOSVQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.258976404 (359h30m43.721068887s) [noise=true , address=45.91.116.8:4048 , piece_id=F56BMKSENGC2HXMUOATBI3GSNPPCVVCCVEEQEZVEDI7QSTVE2YHQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.259581405 (359h30m43.721673898s) [noise=true , address=45.91.116.4:4938 , piece_id=IAHBMRVV72YARM5PTCTUK3LBNLPH6ME6EZQ43EW76UBCEQK5N5RA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.260322587 (359h30m43.72241507s) [noise=true , address=77.90.31.19:4842 , piece_id=BTCUIS4IDUF5M4JA4HU6NIE3Q5LRX7ORPAM6XB534VPPARXLT7LQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.260930324 (359h30m43.723022807s) [noise=true , address=145.79.227.5:4223 , piece_id=O2VCLWGDNEN4C57VGXSL6QTC7Q676LALQI5X2CWPWFZBEN3XAHLA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.261489258 (359h30m43.723581741s) [noise=true , address=145.79.227.4:4187 , piece_id=BW3KBLEUESGC6EBPZPI3GJ3XZYAENVWANQT6JFHHPWCUKSKHTA5Q, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.262275075 (359h30m43.724367548s) [noise=true , address=145.79.227.3:4150 , piece_id=QG7IGRYT5SSGZI3A7FZGJ2DXAJU5RQVZSDBJR3BUQCV76NOPPQ6A, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:59.950872797 (359h30m45.41296528s) [upload=316311]: Next returned nil segment Oct 8 13:01:59.950896232 (359h30m45.412988725s) [upload=316311]: Waiting for error group managing segments... Oct 8 13:02:00.230315202 (359h30m45.692407685s) [noise=true , address=145.79.227.4:4187 , piece_id=BW3KBLEUESGC6EBPZPI3GJ3XZYAENVWANQT6JFHHPWCUKSKHTA5Q, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.249571069 (359h30m45.711663542s) [noise=true , address=145.79.227.5:4223 , piece_id=O2VCLWGDNEN4C57VGXSL6QTC7Q676LALQI5X2CWPWFZBEN3XAHLA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.255522231 (359h30m45.717614724s) [noise=true , address=145.79.227.3:4150 , piece_id=QG7IGRYT5SSGZI3A7FZGJ2DXAJU5RQVZSDBJR3BUQCV76NOPPQ6A, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.259550322 (359h30m45.721642805s) [noise=true , address=145.79.227.18:4005 , piece_id=6IB6TFKCAM7BEUQAMJESVPYV4HNGFZXV7R5YBT5NOM3IJKRBIRPQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.261539359 (359h30m45.723631832s) [noise=true , address=145.79.227.13:4804 , piece_id=5CIOSYYYG2YEYYDM44JAGWEONLRJU2VJOCR6SA3TIIRB4DCBOSVQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.455584772 (359h30m45.917677255s) [noise=true , address=45.91.116.6:4973 , piece_id=GW7S6KTE5DLEUMUAEJE27VOFZY5YSTLOPJRCMKRZJCO3WCTQIJEQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.526039842 (359h30m45.988132325s) [noise=true , address=77.90.31.10:4519 , piece_id=GVBAGJMLRINBGUGLR7K5U7U5UDAUO2TDHVQU3YOGH4LUMOZUJP4A, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.533295910 (359h30m45.995388403s) [noise=true , address=77.90.31.19:4842 , piece_id=BTCUIS4IDUF5M4JA4HU6NIE3Q5LRX7ORPAM6XB534VPPARXLT7LQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.542682743 (359h30m46.004775216s) [noise=true , address=45.91.116.14:4191 , piece_id=OB5O5DHWAVFLR64FIY2JMETOA6YBVHQZ7X623XG37FISJBLWQKKQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.613214860 (359h30m46.075307343s) [noise=true , address=45.91.116.8:4048 , piece_id=F56BMKSENGC2HXMUOATBI3GSNPPCVVCCVEEQEZVEDI7QSTVE2YHQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.820474816 (359h30m46.282567299s) [noise=true , address=45.91.116.6:4975 , piece_id=P5JV6USSWDB335IRLXXCWMJEHYBUDN6KKCYSTJRWDLUEJEOY7RRA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.831238041 (359h30m46.293330524s) [noise=true , address=45.91.116.7:4009 , piece_id=HUHKWUSQB5LGFWPQZHHS6NTISCESL4WCZ5E5XJCDHK55FBWB6XXA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.849605917 (359h30m46.3116984s) [noise=true , address=45.91.116.12:4225 , piece_id=6WTRWQ625W6WUV2DS27PNLPEWPPSR52H3YFOT6SCMG6TXP6W3J2A, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.849649890 (359h30m46.311742373s) [seg_pos={0 0}, upload=316311]: Segment reached optimal threshold of 13 pieces. Oct 8 13:02:00.849835444 (359h30m46.311927917s) [noise=true , address=77.90.31.17:4771 , piece_id=3PXQBNPKNSPGWX457DDSCF7GTKENAKTPLOPHE22LLNFBT5SAZMRQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:44096->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:44096->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44096->77.90.31.17:4771: use of closed network connection Oct 8 13:02:00.849844051 (359h30m46.311936524s) [noise=true , address=77.90.31.17:4771 , piece_id=3PXQBNPKNSPGWX457DDSCF7GTKENAKTPLOPHE22LLNFBT5SAZMRQ, seg_pos={0 0}, upload=316311]: Long tail cancellation due to optimal threshold Oct 8 13:02:00.849879067 (359h30m46.31197154s) [noise=true , address=45.91.116.4:4938 , piece_id=IAHBMRVV72YARM5PTCTUK3LBNLPH6ME6EZQ43EW76UBCEQK5N5RA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Oct 8 13:02:00.849888184 (359h30m46.311980657s) [noise=true , address=45.91.116.4:4938 , piece_id=IAHBMRVV72YARM5PTCTUK3LBNLPH6ME6EZQ43EW76UBCEQK5N5RA, seg_pos={0 0}, upload=316311]: Long tail cancellation due to optimal threshold Oct 8 13:02:00.849927940 (359h30m46.312020413s) [noise=true , address=45.91.116.8:4045 , piece_id=RH3GWGTRTNKP2BZWOPUQSB75VRK63CPWKYJBGQ5PUYACKF46BGRQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Oct 8 13:02:00.849934363 (359h30m46.312026836s) [noise=true , address=45.91.116.8:4045 , piece_id=RH3GWGTRTNKP2BZWOPUQSB75VRK63CPWKYJBGQ5PUYACKF46BGRQ, seg_pos={0 0}, upload=316311]: Long tail cancellation due to optimal threshold Oct 8 13:02:00.849955342 (359h30m46.312047825s) [seg_pos={0 0}, upload=316311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:02:00.850025265 (359h30m46.312117748s) [upload=316311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:02:00.850030355 (359h30m46.312122828s) [upload=316311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:02:01.159121227 (359h30m46.62121371s) [upload=316311]: Done uploading segments... Oct 8 13:02:01.159145763 (359h30m46.621238246s) [upload=316311]: Upload finished. err: <nil> Duration: 30.001696573s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.053678582s FAIL (2025-10-08 13:01:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.053708628s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....X........X..X..X....XX..XXX.XX..X.XX..X........XX..XX...] duration: 6.45727647s FAIL (2025-10-08 13:01:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 4VAARCNHICUT6M7MNE2CLDOE7COG3U5IQ25VNZOPHOERWZ4BXQWQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: D27PLH63BME3B7SXB2JSIFXDZBU5SULATZH73LPSWKICVQQ3S43A) context deadline exceeded ======LOGS====== Oct 8 12:58:50.905013700 (359h27m36.367106183s) [upload=316262]: Starting upload Oct 8 12:58:50.905097930 (359h27m36.367190413s) [upload=316262]: Done starting upload Oct 8 12:58:50.905220854 (359h27m36.367313337s) [upload=316262]: Uploading segments... Oct 8 12:58:50.906644945 (359h27m36.368737428s) [upload=316262]: Got next segment. Inline: false Oct 8 12:58:50.906779732 (359h27m36.368872225s) [upload=316262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:58:50.906786285 (359h27m36.368878758s) [upload=316262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:58:51.168160945 (359h27m36.630253429s) [seg_pos={0 0}, upload=316262]: Begin upload segment... Oct 8 12:58:51.168427833 (359h27m36.630520316s) [noise=true , address=145.79.227.4:4187 , piece_id=735BUB6LD356QOUX67TEREW3WH6OHPLPTS6RJQBOVSVIXES6NO2A, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.168508517 (359h27m36.63060103s) [seg_pos={0 0}, upload=316262]: Done begin upload segment. Oct 8 12:58:51.168595272 (359h27m36.630687755s) [noise=true , address=45.91.116.14:4191 , piece_id=PVMS6GNKYMWPJFHS7QSO2HE3YXNZDSB65F3P4XEXV4VYWTFDPQEQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.169478023 (359h27m36.631570496s) [noise=true , address=45.91.116.6:4975 , piece_id=SYTAOBSBAKYWFOXNQWVUMB3UPS3VFIYMUX54C4UBWDKZE3TFAHZQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.169690969 (359h27m36.631783452s) [noise=true , address=45.91.116.13:4154 , piece_id=NX7N3FKWZWV4HN2I77CL4DHNJ6ZPXAP2FAODVTUO2552ZEF5B43A, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.170477706 (359h27m36.632570179s) [noise=true , address=45.91.116.12:4227 , piece_id=5RC6EVOMUKDJLHAE7BGIVT7EIN23P6GJVRVYFHYLQPLW5KDP3SZQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.171188611 (359h27m36.633281084s) [noise=true , address=45.91.116.12:4225 , piece_id=D27PLH63BME3B7SXB2JSIFXDZBU5SULATZH73LPSWKICVQQ3S43A, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.173666429 (359h27m36.635758902s) [noise=true , address=45.91.116.7:4009 , piece_id=FM5TRV2QHP6N2NSBN7ZTZEKCRNLCQ7BCIQLNVLGQNCK3P5EH7HEQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.174268565 (359h27m36.636361048s) [noise=true , address=145.79.227.6:4258 , piece_id=ZEKTR2HS2Z6YSDX72XGESSSGHWGWDPGIVWMWBDUVGPUN4QKPAZ3A, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.174490968 (359h27m36.636583451s) [noise=true , address=145.79.227.11:4731 , piece_id=YUY3RC2DEPHJ3TAL364RKKDA3XMCQC3OWQWATL2EFLU5WHBVA5GA, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.175093466 (359h27m36.637185939s) [noise=true , address=77.90.31.13:4626 , piece_id=CY5OQBVPCQE7GON6XDUBA5WAM4KAA3ZEYBO4SXZ4ATQ5C74JR6OA, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.175139754 (359h27m36.637232227s) [noise=true , address=45.91.116.6:4974 , piece_id=JQYTQTT5BSHCVIV563BKRQ6RKJCSZWFH5573CKW7BMZKBPGEHHPQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.175636830 (359h27m36.637729313s) [noise=true , address=145.79.227.3:4151 , piece_id=GNVJR3HJNNXD76Q4QTMPH5MJDVWLL5GU6ZQNFGG5E4M4VEASA2KQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.175886947 (359h27m36.63797943s) [noise=true , address=145.79.227.5:4222 , piece_id=VID5VHIW3OYCSHWBEC5JGBEIQLQTNVSBO4Q4E65NYNI6A3MEIPFQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.176270037 (359h27m36.63836251s) [noise=true , address=145.79.227.14:4837 , piece_id=4YZDOIME7VI53LL5H35N64FPXB4FODEKROQ6R43AVDKRS6F2RXDA, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.177113232 (359h27m36.639205705s) [noise=true , address=45.91.116.4:4939 , piece_id=MWKWCCIKHUMKUJGPDE7WGJSIVHV3HAP6PE4SSGCRQ5QYR6VUFXOQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.177924817 (359h27m36.64001729s) [noise=true , address=45.91.116.12:4226 , piece_id=4VAARCNHICUT6M7MNE2CLDOE7COG3U5IQ25VNZOPHOERWZ4BXQWQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:52.893320449 (359h27m38.355412982s) [upload=316262]: Next returned nil segment Oct 8 12:58:52.893347439 (359h27m38.355439922s) [upload=316262]: Waiting for error group managing segments... Oct 8 12:58:53.172715262 (359h27m38.634807735s) [noise=true , address=145.79.227.11:4731 , piece_id=YUY3RC2DEPHJ3TAL364RKKDA3XMCQC3OWQWATL2EFLU5WHBVA5GA, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.175288112 (359h27m38.637380585s) [noise=true , address=145.79.227.6:4258 , piece_id=ZEKTR2HS2Z6YSDX72XGESSSGHWGWDPGIVWMWBDUVGPUN4QKPAZ3A, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.175745963 (359h27m38.637838436s) [noise=true , address=145.79.227.3:4151 , piece_id=GNVJR3HJNNXD76Q4QTMPH5MJDVWLL5GU6ZQNFGG5E4M4VEASA2KQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.177270717 (359h27m38.6393632s) [noise=true , address=145.79.227.14:4837 , piece_id=4YZDOIME7VI53LL5H35N64FPXB4FODEKROQ6R43AVDKRS6F2RXDA, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.180782294 (359h27m38.642874777s) [noise=true , address=145.79.227.4:4187 , piece_id=735BUB6LD356QOUX67TEREW3WH6OHPLPTS6RJQBOVSVIXES6NO2A, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.198430931 (359h27m38.660523404s) [noise=true , address=145.79.227.5:4222 , piece_id=VID5VHIW3OYCSHWBEC5JGBEIQLQTNVSBO4Q4E65NYNI6A3MEIPFQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.410582530 (359h27m38.872675013s) [noise=true , address=45.91.116.6:4975 , piece_id=SYTAOBSBAKYWFOXNQWVUMB3UPS3VFIYMUX54C4UBWDKZE3TFAHZQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.411076881 (359h27m38.873169354s) [noise=true , address=45.91.116.12:4225 , piece_id=D27PLH63BME3B7SXB2JSIFXDZBU5SULATZH73LPSWKICVQQ3S43A, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.423225625 (359h27m38.885318098s) [noise=true , address=45.91.116.12:4226 , piece_id=4VAARCNHICUT6M7MNE2CLDOE7COG3U5IQ25VNZOPHOERWZ4BXQWQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.454938643 (359h27m38.917031126s) [noise=true , address=45.91.116.7:4009 , piece_id=FM5TRV2QHP6N2NSBN7ZTZEKCRNLCQ7BCIQLNVLGQNCK3P5EH7HEQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.458700767 (359h27m38.92079324s) [noise=true , address=45.91.116.13:4154 , piece_id=NX7N3FKWZWV4HN2I77CL4DHNJ6ZPXAP2FAODVTUO2552ZEF5B43A, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.478635286 (359h27m38.940727759s) [noise=true , address=45.91.116.4:4939 , piece_id=MWKWCCIKHUMKUJGPDE7WGJSIVHV3HAP6PE4SSGCRQ5QYR6VUFXOQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.488485903 (359h27m38.950578386s) [noise=true , address=45.91.116.14:4191 , piece_id=PVMS6GNKYMWPJFHS7QSO2HE3YXNZDSB65F3P4XEXV4VYWTFDPQEQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.488505601 (359h27m38.950598084s) [seg_pos={0 0}, upload=316262]: Segment reached optimal threshold of 13 pieces. Oct 8 12:58:53.488615380 (359h27m38.950707893s) [noise=true , address=45.91.116.6:4974 , piece_id=JQYTQTT5BSHCVIV563BKRQ6RKJCSZWFH5573CKW7BMZKBPGEHHPQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 8 12:58:53.488629176 (359h27m38.950721689s) [noise=true , address=45.91.116.6:4974 , piece_id=JQYTQTT5BSHCVIV563BKRQ6RKJCSZWFH5573CKW7BMZKBPGEHHPQ, seg_pos={0 0}, upload=316262]: Long tail cancellation due to optimal threshold Oct 8 12:58:53.488761809 (359h27m38.950854292s) [noise=true , address=77.90.31.13:4626 , piece_id=CY5OQBVPCQE7GON6XDUBA5WAM4KAA3ZEYBO4SXZ4ATQ5C74JR6OA, seg_pos={0 0}, upload=316262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Oct 8 12:58:53.488769443 (359h27m38.950861916s) [noise=true , address=77.90.31.13:4626 , piece_id=CY5OQBVPCQE7GON6XDUBA5WAM4KAA3ZEYBO4SXZ4ATQ5C74JR6OA, seg_pos={0 0}, upload=316262]: Long tail cancellation due to optimal threshold Oct 8 12:58:53.488839506 (359h27m38.950931989s) [noise=true , address=45.91.116.12:4227 , piece_id=5RC6EVOMUKDJLHAE7BGIVT7EIN23P6GJVRVYFHYLQPLW5KDP3SZQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Oct 8 12:58:53.488845268 (359h27m38.950937751s) [noise=true , address=45.91.116.12:4227 , piece_id=5RC6EVOMUKDJLHAE7BGIVT7EIN23P6GJVRVYFHYLQPLW5KDP3SZQ, seg_pos={0 0}, upload=316262]: Long tail cancellation due to optimal threshold Oct 8 12:58:53.488861008 (359h27m38.950953481s) [seg_pos={0 0}, upload=316262]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:58:53.488923888 (359h27m38.951016361s) [upload=316262]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:58:53.488928937 (359h27m38.95102142s) [upload=316262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:58:53.875015854 (359h27m39.337108337s) [upload=316262]: Done uploading segments... Oct 8 12:58:53.875040161 (359h27m39.337132644s) [upload=316262]: Upload finished. err: <nil> Duration: 6.457312408s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.....X........X..X..X....XX..XXX.XX..X.XX..X........XX..XX..] duration: 27.271115744s FAIL (2025-10-08 13:00:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 4VAARCNHICUT6M7MNE2CLDOE7COG3U5IQ25VNZOPHOERWZ4BXQWQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: D27PLH63BME3B7SXB2JSIFXDZBU5SULATZH73LPSWKICVQQ3S43A) context deadline exceeded ======LOGS====== Oct 8 12:58:50.905013700 (359h27m36.367106183s) [upload=316262]: Starting upload Oct 8 12:58:50.905097930 (359h27m36.367190413s) [upload=316262]: Done starting upload Oct 8 12:58:50.905220854 (359h27m36.367313337s) [upload=316262]: Uploading segments... Oct 8 12:58:50.906644945 (359h27m36.368737428s) [upload=316262]: Got next segment. Inline: false Oct 8 12:58:50.906779732 (359h27m36.368872225s) [upload=316262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:58:50.906786285 (359h27m36.368878758s) [upload=316262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:58:51.168160945 (359h27m36.630253429s) [seg_pos={0 0}, upload=316262]: Begin upload segment... Oct 8 12:58:51.168427833 (359h27m36.630520316s) [noise=true , address=145.79.227.4:4187 , piece_id=735BUB6LD356QOUX67TEREW3WH6OHPLPTS6RJQBOVSVIXES6NO2A, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.168508517 (359h27m36.63060103s) [seg_pos={0 0}, upload=316262]: Done begin upload segment. Oct 8 12:58:51.168595272 (359h27m36.630687755s) [noise=true , address=45.91.116.14:4191 , piece_id=PVMS6GNKYMWPJFHS7QSO2HE3YXNZDSB65F3P4XEXV4VYWTFDPQEQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.169478023 (359h27m36.631570496s) [noise=true , address=45.91.116.6:4975 , piece_id=SYTAOBSBAKYWFOXNQWVUMB3UPS3VFIYMUX54C4UBWDKZE3TFAHZQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.169690969 (359h27m36.631783452s) [noise=true , address=45.91.116.13:4154 , piece_id=NX7N3FKWZWV4HN2I77CL4DHNJ6ZPXAP2FAODVTUO2552ZEF5B43A, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.170477706 (359h27m36.632570179s) [noise=true , address=45.91.116.12:4227 , piece_id=5RC6EVOMUKDJLHAE7BGIVT7EIN23P6GJVRVYFHYLQPLW5KDP3SZQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.171188611 (359h27m36.633281084s) [noise=true , address=45.91.116.12:4225 , piece_id=D27PLH63BME3B7SXB2JSIFXDZBU5SULATZH73LPSWKICVQQ3S43A, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.173666429 (359h27m36.635758902s) [noise=true , address=45.91.116.7:4009 , piece_id=FM5TRV2QHP6N2NSBN7ZTZEKCRNLCQ7BCIQLNVLGQNCK3P5EH7HEQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.174268565 (359h27m36.636361048s) [noise=true , address=145.79.227.6:4258 , piece_id=ZEKTR2HS2Z6YSDX72XGESSSGHWGWDPGIVWMWBDUVGPUN4QKPAZ3A, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.174490968 (359h27m36.636583451s) [noise=true , address=145.79.227.11:4731 , piece_id=YUY3RC2DEPHJ3TAL364RKKDA3XMCQC3OWQWATL2EFLU5WHBVA5GA, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.175093466 (359h27m36.637185939s) [noise=true , address=77.90.31.13:4626 , piece_id=CY5OQBVPCQE7GON6XDUBA5WAM4KAA3ZEYBO4SXZ4ATQ5C74JR6OA, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.175139754 (359h27m36.637232227s) [noise=true , address=45.91.116.6:4974 , piece_id=JQYTQTT5BSHCVIV563BKRQ6RKJCSZWFH5573CKW7BMZKBPGEHHPQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.175636830 (359h27m36.637729313s) [noise=true , address=145.79.227.3:4151 , piece_id=GNVJR3HJNNXD76Q4QTMPH5MJDVWLL5GU6ZQNFGG5E4M4VEASA2KQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.175886947 (359h27m36.63797943s) [noise=true , address=145.79.227.5:4222 , piece_id=VID5VHIW3OYCSHWBEC5JGBEIQLQTNVSBO4Q4E65NYNI6A3MEIPFQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.176270037 (359h27m36.63836251s) [noise=true , address=145.79.227.14:4837 , piece_id=4YZDOIME7VI53LL5H35N64FPXB4FODEKROQ6R43AVDKRS6F2RXDA, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.177113232 (359h27m36.639205705s) [noise=true , address=45.91.116.4:4939 , piece_id=MWKWCCIKHUMKUJGPDE7WGJSIVHV3HAP6PE4SSGCRQ5QYR6VUFXOQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.177924817 (359h27m36.64001729s) [noise=true , address=45.91.116.12:4226 , piece_id=4VAARCNHICUT6M7MNE2CLDOE7COG3U5IQ25VNZOPHOERWZ4BXQWQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:52.893320449 (359h27m38.355412982s) [upload=316262]: Next returned nil segment Oct 8 12:58:52.893347439 (359h27m38.355439922s) [upload=316262]: Waiting for error group managing segments... Oct 8 12:58:53.172715262 (359h27m38.634807735s) [noise=true , address=145.79.227.11:4731 , piece_id=YUY3RC2DEPHJ3TAL364RKKDA3XMCQC3OWQWATL2EFLU5WHBVA5GA, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.175288112 (359h27m38.637380585s) [noise=true , address=145.79.227.6:4258 , piece_id=ZEKTR2HS2Z6YSDX72XGESSSGHWGWDPGIVWMWBDUVGPUN4QKPAZ3A, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.175745963 (359h27m38.637838436s) [noise=true , address=145.79.227.3:4151 , piece_id=GNVJR3HJNNXD76Q4QTMPH5MJDVWLL5GU6ZQNFGG5E4M4VEASA2KQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.177270717 (359h27m38.6393632s) [noise=true , address=145.79.227.14:4837 , piece_id=4YZDOIME7VI53LL5H35N64FPXB4FODEKROQ6R43AVDKRS6F2RXDA, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.180782294 (359h27m38.642874777s) [noise=true , address=145.79.227.4:4187 , piece_id=735BUB6LD356QOUX67TEREW3WH6OHPLPTS6RJQBOVSVIXES6NO2A, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.198430931 (359h27m38.660523404s) [noise=true , address=145.79.227.5:4222 , piece_id=VID5VHIW3OYCSHWBEC5JGBEIQLQTNVSBO4Q4E65NYNI6A3MEIPFQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.410582530 (359h27m38.872675013s) [noise=true , address=45.91.116.6:4975 , piece_id=SYTAOBSBAKYWFOXNQWVUMB3UPS3VFIYMUX54C4UBWDKZE3TFAHZQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.411076881 (359h27m38.873169354s) [noise=true , address=45.91.116.12:4225 , piece_id=D27PLH63BME3B7SXB2JSIFXDZBU5SULATZH73LPSWKICVQQ3S43A, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.423225625 (359h27m38.885318098s) [noise=true , address=45.91.116.12:4226 , piece_id=4VAARCNHICUT6M7MNE2CLDOE7COG3U5IQ25VNZOPHOERWZ4BXQWQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.454938643 (359h27m38.917031126s) [noise=true , address=45.91.116.7:4009 , piece_id=FM5TRV2QHP6N2NSBN7ZTZEKCRNLCQ7BCIQLNVLGQNCK3P5EH7HEQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.458700767 (359h27m38.92079324s) [noise=true , address=45.91.116.13:4154 , piece_id=NX7N3FKWZWV4HN2I77CL4DHNJ6ZPXAP2FAODVTUO2552ZEF5B43A, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.478635286 (359h27m38.940727759s) [noise=true , address=45.91.116.4:4939 , piece_id=MWKWCCIKHUMKUJGPDE7WGJSIVHV3HAP6PE4SSGCRQ5QYR6VUFXOQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.488485903 (359h27m38.950578386s) [noise=true , address=45.91.116.14:4191 , piece_id=PVMS6GNKYMWPJFHS7QSO2HE3YXNZDSB65F3P4XEXV4VYWTFDPQEQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.488505601 (359h27m38.950598084s) [seg_pos={0 0}, upload=316262]: Segment reached optimal threshold of 13 pieces. Oct 8 12:58:53.488615380 (359h27m38.950707893s) [noise=true , address=45.91.116.6:4974 , piece_id=JQYTQTT5BSHCVIV563BKRQ6RKJCSZWFH5573CKW7BMZKBPGEHHPQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 8 12:58:53.488629176 (359h27m38.950721689s) [noise=true , address=45.91.116.6:4974 , piece_id=JQYTQTT5BSHCVIV563BKRQ6RKJCSZWFH5573CKW7BMZKBPGEHHPQ, seg_pos={0 0}, upload=316262]: Long tail cancellation due to optimal threshold Oct 8 12:58:53.488761809 (359h27m38.950854292s) [noise=true , address=77.90.31.13:4626 , piece_id=CY5OQBVPCQE7GON6XDUBA5WAM4KAA3ZEYBO4SXZ4ATQ5C74JR6OA, seg_pos={0 0}, upload=316262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Oct 8 12:58:53.488769443 (359h27m38.950861916s) [noise=true , address=77.90.31.13:4626 , piece_id=CY5OQBVPCQE7GON6XDUBA5WAM4KAA3ZEYBO4SXZ4ATQ5C74JR6OA, seg_pos={0 0}, upload=316262]: Long tail cancellation due to optimal threshold Oct 8 12:58:53.488839506 (359h27m38.950931989s) [noise=true , address=45.91.116.12:4227 , piece_id=5RC6EVOMUKDJLHAE7BGIVT7EIN23P6GJVRVYFHYLQPLW5KDP3SZQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Oct 8 12:58:53.488845268 (359h27m38.950937751s) [noise=true , address=45.91.116.12:4227 , piece_id=5RC6EVOMUKDJLHAE7BGIVT7EIN23P6GJVRVYFHYLQPLW5KDP3SZQ, seg_pos={0 0}, upload=316262]: Long tail cancellation due to optimal threshold Oct 8 12:58:53.488861008 (359h27m38.950953481s) [seg_pos={0 0}, upload=316262]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:58:53.488923888 (359h27m38.951016361s) [upload=316262]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:58:53.488928937 (359h27m38.95102142s) [upload=316262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:58:53.875015854 (359h27m39.337108337s) [upload=316262]: Done uploading segments... Oct 8 12:58:53.875040161 (359h27m39.337132644s) [upload=316262]: Upload finished. err: <nil> Duration: 27.271269057s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.354348454s FAIL (2025-10-08 13:00:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.35437761s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.341008766s FAIL (2025-10-08 12:59:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.34111056s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [......X........X..X..X....XX..XXX.XX..X.XX..X........XX..XX.] duration: 6.972543466s FAIL (2025-10-08 12:59:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 4VAARCNHICUT6M7MNE2CLDOE7COG3U5IQ25VNZOPHOERWZ4BXQWQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: D27PLH63BME3B7SXB2JSIFXDZBU5SULATZH73LPSWKICVQQ3S43A) context deadline exceeded ======LOGS====== Oct 8 12:58:50.905013700 (359h27m36.367106183s) [upload=316262]: Starting upload Oct 8 12:58:50.905097930 (359h27m36.367190413s) [upload=316262]: Done starting upload Oct 8 12:58:50.905220854 (359h27m36.367313337s) [upload=316262]: Uploading segments... Oct 8 12:58:50.906644945 (359h27m36.368737428s) [upload=316262]: Got next segment. Inline: false Oct 8 12:58:50.906779732 (359h27m36.368872225s) [upload=316262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:58:50.906786285 (359h27m36.368878758s) [upload=316262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:58:51.168160945 (359h27m36.630253429s) [seg_pos={0 0}, upload=316262]: Begin upload segment... Oct 8 12:58:51.168427833 (359h27m36.630520316s) [noise=true , address=145.79.227.4:4187 , piece_id=735BUB6LD356QOUX67TEREW3WH6OHPLPTS6RJQBOVSVIXES6NO2A, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.168508517 (359h27m36.63060103s) [seg_pos={0 0}, upload=316262]: Done begin upload segment. Oct 8 12:58:51.168595272 (359h27m36.630687755s) [noise=true , address=45.91.116.14:4191 , piece_id=PVMS6GNKYMWPJFHS7QSO2HE3YXNZDSB65F3P4XEXV4VYWTFDPQEQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.169478023 (359h27m36.631570496s) [noise=true , address=45.91.116.6:4975 , piece_id=SYTAOBSBAKYWFOXNQWVUMB3UPS3VFIYMUX54C4UBWDKZE3TFAHZQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.169690969 (359h27m36.631783452s) [noise=true , address=45.91.116.13:4154 , piece_id=NX7N3FKWZWV4HN2I77CL4DHNJ6ZPXAP2FAODVTUO2552ZEF5B43A, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.170477706 (359h27m36.632570179s) [noise=true , address=45.91.116.12:4227 , piece_id=5RC6EVOMUKDJLHAE7BGIVT7EIN23P6GJVRVYFHYLQPLW5KDP3SZQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.171188611 (359h27m36.633281084s) [noise=true , address=45.91.116.12:4225 , piece_id=D27PLH63BME3B7SXB2JSIFXDZBU5SULATZH73LPSWKICVQQ3S43A, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.173666429 (359h27m36.635758902s) [noise=true , address=45.91.116.7:4009 , piece_id=FM5TRV2QHP6N2NSBN7ZTZEKCRNLCQ7BCIQLNVLGQNCK3P5EH7HEQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.174268565 (359h27m36.636361048s) [noise=true , address=145.79.227.6:4258 , piece_id=ZEKTR2HS2Z6YSDX72XGESSSGHWGWDPGIVWMWBDUVGPUN4QKPAZ3A, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.174490968 (359h27m36.636583451s) [noise=true , address=145.79.227.11:4731 , piece_id=YUY3RC2DEPHJ3TAL364RKKDA3XMCQC3OWQWATL2EFLU5WHBVA5GA, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.175093466 (359h27m36.637185939s) [noise=true , address=77.90.31.13:4626 , piece_id=CY5OQBVPCQE7GON6XDUBA5WAM4KAA3ZEYBO4SXZ4ATQ5C74JR6OA, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.175139754 (359h27m36.637232227s) [noise=true , address=45.91.116.6:4974 , piece_id=JQYTQTT5BSHCVIV563BKRQ6RKJCSZWFH5573CKW7BMZKBPGEHHPQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.175636830 (359h27m36.637729313s) [noise=true , address=145.79.227.3:4151 , piece_id=GNVJR3HJNNXD76Q4QTMPH5MJDVWLL5GU6ZQNFGG5E4M4VEASA2KQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.175886947 (359h27m36.63797943s) [noise=true , address=145.79.227.5:4222 , piece_id=VID5VHIW3OYCSHWBEC5JGBEIQLQTNVSBO4Q4E65NYNI6A3MEIPFQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.176270037 (359h27m36.63836251s) [noise=true , address=145.79.227.14:4837 , piece_id=4YZDOIME7VI53LL5H35N64FPXB4FODEKROQ6R43AVDKRS6F2RXDA, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.177113232 (359h27m36.639205705s) [noise=true , address=45.91.116.4:4939 , piece_id=MWKWCCIKHUMKUJGPDE7WGJSIVHV3HAP6PE4SSGCRQ5QYR6VUFXOQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.177924817 (359h27m36.64001729s) [noise=true , address=45.91.116.12:4226 , piece_id=4VAARCNHICUT6M7MNE2CLDOE7COG3U5IQ25VNZOPHOERWZ4BXQWQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:52.893320449 (359h27m38.355412982s) [upload=316262]: Next returned nil segment Oct 8 12:58:52.893347439 (359h27m38.355439922s) [upload=316262]: Waiting for error group managing segments... Oct 8 12:58:53.172715262 (359h27m38.634807735s) [noise=true , address=145.79.227.11:4731 , piece_id=YUY3RC2DEPHJ3TAL364RKKDA3XMCQC3OWQWATL2EFLU5WHBVA5GA, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.175288112 (359h27m38.637380585s) [noise=true , address=145.79.227.6:4258 , piece_id=ZEKTR2HS2Z6YSDX72XGESSSGHWGWDPGIVWMWBDUVGPUN4QKPAZ3A, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.175745963 (359h27m38.637838436s) [noise=true , address=145.79.227.3:4151 , piece_id=GNVJR3HJNNXD76Q4QTMPH5MJDVWLL5GU6ZQNFGG5E4M4VEASA2KQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.177270717 (359h27m38.6393632s) [noise=true , address=145.79.227.14:4837 , piece_id=4YZDOIME7VI53LL5H35N64FPXB4FODEKROQ6R43AVDKRS6F2RXDA, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.180782294 (359h27m38.642874777s) [noise=true , address=145.79.227.4:4187 , piece_id=735BUB6LD356QOUX67TEREW3WH6OHPLPTS6RJQBOVSVIXES6NO2A, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.198430931 (359h27m38.660523404s) [noise=true , address=145.79.227.5:4222 , piece_id=VID5VHIW3OYCSHWBEC5JGBEIQLQTNVSBO4Q4E65NYNI6A3MEIPFQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.410582530 (359h27m38.872675013s) [noise=true , address=45.91.116.6:4975 , piece_id=SYTAOBSBAKYWFOXNQWVUMB3UPS3VFIYMUX54C4UBWDKZE3TFAHZQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.411076881 (359h27m38.873169354s) [noise=true , address=45.91.116.12:4225 , piece_id=D27PLH63BME3B7SXB2JSIFXDZBU5SULATZH73LPSWKICVQQ3S43A, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.423225625 (359h27m38.885318098s) [noise=true , address=45.91.116.12:4226 , piece_id=4VAARCNHICUT6M7MNE2CLDOE7COG3U5IQ25VNZOPHOERWZ4BXQWQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.454938643 (359h27m38.917031126s) [noise=true , address=45.91.116.7:4009 , piece_id=FM5TRV2QHP6N2NSBN7ZTZEKCRNLCQ7BCIQLNVLGQNCK3P5EH7HEQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.458700767 (359h27m38.92079324s) [noise=true , address=45.91.116.13:4154 , piece_id=NX7N3FKWZWV4HN2I77CL4DHNJ6ZPXAP2FAODVTUO2552ZEF5B43A, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.478635286 (359h27m38.940727759s) [noise=true , address=45.91.116.4:4939 , piece_id=MWKWCCIKHUMKUJGPDE7WGJSIVHV3HAP6PE4SSGCRQ5QYR6VUFXOQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.488485903 (359h27m38.950578386s) [noise=true , address=45.91.116.14:4191 , piece_id=PVMS6GNKYMWPJFHS7QSO2HE3YXNZDSB65F3P4XEXV4VYWTFDPQEQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.488505601 (359h27m38.950598084s) [seg_pos={0 0}, upload=316262]: Segment reached optimal threshold of 13 pieces. Oct 8 12:58:53.488615380 (359h27m38.950707893s) [noise=true , address=45.91.116.6:4974 , piece_id=JQYTQTT5BSHCVIV563BKRQ6RKJCSZWFH5573CKW7BMZKBPGEHHPQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 8 12:58:53.488629176 (359h27m38.950721689s) [noise=true , address=45.91.116.6:4974 , piece_id=JQYTQTT5BSHCVIV563BKRQ6RKJCSZWFH5573CKW7BMZKBPGEHHPQ, seg_pos={0 0}, upload=316262]: Long tail cancellation due to optimal threshold Oct 8 12:58:53.488761809 (359h27m38.950854292s) [noise=true , address=77.90.31.13:4626 , piece_id=CY5OQBVPCQE7GON6XDUBA5WAM4KAA3ZEYBO4SXZ4ATQ5C74JR6OA, seg_pos={0 0}, upload=316262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Oct 8 12:58:53.488769443 (359h27m38.950861916s) [noise=true , address=77.90.31.13:4626 , piece_id=CY5OQBVPCQE7GON6XDUBA5WAM4KAA3ZEYBO4SXZ4ATQ5C74JR6OA, seg_pos={0 0}, upload=316262]: Long tail cancellation due to optimal threshold Oct 8 12:58:53.488839506 (359h27m38.950931989s) [noise=true , address=45.91.116.12:4227 , piece_id=5RC6EVOMUKDJLHAE7BGIVT7EIN23P6GJVRVYFHYLQPLW5KDP3SZQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Oct 8 12:58:53.488845268 (359h27m38.950937751s) [noise=true , address=45.91.116.12:4227 , piece_id=5RC6EVOMUKDJLHAE7BGIVT7EIN23P6GJVRVYFHYLQPLW5KDP3SZQ, seg_pos={0 0}, upload=316262]: Long tail cancellation due to optimal threshold Oct 8 12:58:53.488861008 (359h27m38.950953481s) [seg_pos={0 0}, upload=316262]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:58:53.488923888 (359h27m38.951016361s) [upload=316262]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:58:53.488928937 (359h27m38.95102142s) [upload=316262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:58:53.875015854 (359h27m39.337108337s) [upload=316262]: Done uploading segments... Oct 8 12:58:53.875040161 (359h27m39.337132644s) [upload=316262]: Upload finished. err: <nil> Duration: 6.972588343s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.485168024s FAIL (2025-10-08 12:59:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.485232627s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.......X........X..X..X....XX..XXX.XX..X.XX..X........XX..XX] duration: 30.00189801s FAIL (2025-10-08 12:59:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 4VAARCNHICUT6M7MNE2CLDOE7COG3U5IQ25VNZOPHOERWZ4BXQWQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: D27PLH63BME3B7SXB2JSIFXDZBU5SULATZH73LPSWKICVQQ3S43A) context deadline exceeded ======LOGS====== Oct 8 12:58:50.905013700 (359h27m36.367106183s) [upload=316262]: Starting upload Oct 8 12:58:50.905097930 (359h27m36.367190413s) [upload=316262]: Done starting upload Oct 8 12:58:50.905220854 (359h27m36.367313337s) [upload=316262]: Uploading segments... Oct 8 12:58:50.906644945 (359h27m36.368737428s) [upload=316262]: Got next segment. Inline: false Oct 8 12:58:50.906779732 (359h27m36.368872225s) [upload=316262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:58:50.906786285 (359h27m36.368878758s) [upload=316262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:58:51.168160945 (359h27m36.630253429s) [seg_pos={0 0}, upload=316262]: Begin upload segment... Oct 8 12:58:51.168427833 (359h27m36.630520316s) [noise=true , address=145.79.227.4:4187 , piece_id=735BUB6LD356QOUX67TEREW3WH6OHPLPTS6RJQBOVSVIXES6NO2A, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.168508517 (359h27m36.63060103s) [seg_pos={0 0}, upload=316262]: Done begin upload segment. Oct 8 12:58:51.168595272 (359h27m36.630687755s) [noise=true , address=45.91.116.14:4191 , piece_id=PVMS6GNKYMWPJFHS7QSO2HE3YXNZDSB65F3P4XEXV4VYWTFDPQEQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.169478023 (359h27m36.631570496s) [noise=true , address=45.91.116.6:4975 , piece_id=SYTAOBSBAKYWFOXNQWVUMB3UPS3VFIYMUX54C4UBWDKZE3TFAHZQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.169690969 (359h27m36.631783452s) [noise=true , address=45.91.116.13:4154 , piece_id=NX7N3FKWZWV4HN2I77CL4DHNJ6ZPXAP2FAODVTUO2552ZEF5B43A, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.170477706 (359h27m36.632570179s) [noise=true , address=45.91.116.12:4227 , piece_id=5RC6EVOMUKDJLHAE7BGIVT7EIN23P6GJVRVYFHYLQPLW5KDP3SZQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.171188611 (359h27m36.633281084s) [noise=true , address=45.91.116.12:4225 , piece_id=D27PLH63BME3B7SXB2JSIFXDZBU5SULATZH73LPSWKICVQQ3S43A, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.173666429 (359h27m36.635758902s) [noise=true , address=45.91.116.7:4009 , piece_id=FM5TRV2QHP6N2NSBN7ZTZEKCRNLCQ7BCIQLNVLGQNCK3P5EH7HEQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.174268565 (359h27m36.636361048s) [noise=true , address=145.79.227.6:4258 , piece_id=ZEKTR2HS2Z6YSDX72XGESSSGHWGWDPGIVWMWBDUVGPUN4QKPAZ3A, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.174490968 (359h27m36.636583451s) [noise=true , address=145.79.227.11:4731 , piece_id=YUY3RC2DEPHJ3TAL364RKKDA3XMCQC3OWQWATL2EFLU5WHBVA5GA, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.175093466 (359h27m36.637185939s) [noise=true , address=77.90.31.13:4626 , piece_id=CY5OQBVPCQE7GON6XDUBA5WAM4KAA3ZEYBO4SXZ4ATQ5C74JR6OA, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.175139754 (359h27m36.637232227s) [noise=true , address=45.91.116.6:4974 , piece_id=JQYTQTT5BSHCVIV563BKRQ6RKJCSZWFH5573CKW7BMZKBPGEHHPQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.175636830 (359h27m36.637729313s) [noise=true , address=145.79.227.3:4151 , piece_id=GNVJR3HJNNXD76Q4QTMPH5MJDVWLL5GU6ZQNFGG5E4M4VEASA2KQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.175886947 (359h27m36.63797943s) [noise=true , address=145.79.227.5:4222 , piece_id=VID5VHIW3OYCSHWBEC5JGBEIQLQTNVSBO4Q4E65NYNI6A3MEIPFQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.176270037 (359h27m36.63836251s) [noise=true , address=145.79.227.14:4837 , piece_id=4YZDOIME7VI53LL5H35N64FPXB4FODEKROQ6R43AVDKRS6F2RXDA, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.177113232 (359h27m36.639205705s) [noise=true , address=45.91.116.4:4939 , piece_id=MWKWCCIKHUMKUJGPDE7WGJSIVHV3HAP6PE4SSGCRQ5QYR6VUFXOQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:51.177924817 (359h27m36.64001729s) [noise=true , address=45.91.116.12:4226 , piece_id=4VAARCNHICUT6M7MNE2CLDOE7COG3U5IQ25VNZOPHOERWZ4BXQWQ, seg_pos={0 0}, upload=316262]: Uploading piece... Oct 8 12:58:52.893320449 (359h27m38.355412982s) [upload=316262]: Next returned nil segment Oct 8 12:58:52.893347439 (359h27m38.355439922s) [upload=316262]: Waiting for error group managing segments... Oct 8 12:58:53.172715262 (359h27m38.634807735s) [noise=true , address=145.79.227.11:4731 , piece_id=YUY3RC2DEPHJ3TAL364RKKDA3XMCQC3OWQWATL2EFLU5WHBVA5GA, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.175288112 (359h27m38.637380585s) [noise=true , address=145.79.227.6:4258 , piece_id=ZEKTR2HS2Z6YSDX72XGESSSGHWGWDPGIVWMWBDUVGPUN4QKPAZ3A, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.175745963 (359h27m38.637838436s) [noise=true , address=145.79.227.3:4151 , piece_id=GNVJR3HJNNXD76Q4QTMPH5MJDVWLL5GU6ZQNFGG5E4M4VEASA2KQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.177270717 (359h27m38.6393632s) [noise=true , address=145.79.227.14:4837 , piece_id=4YZDOIME7VI53LL5H35N64FPXB4FODEKROQ6R43AVDKRS6F2RXDA, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.180782294 (359h27m38.642874777s) [noise=true , address=145.79.227.4:4187 , piece_id=735BUB6LD356QOUX67TEREW3WH6OHPLPTS6RJQBOVSVIXES6NO2A, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.198430931 (359h27m38.660523404s) [noise=true , address=145.79.227.5:4222 , piece_id=VID5VHIW3OYCSHWBEC5JGBEIQLQTNVSBO4Q4E65NYNI6A3MEIPFQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.410582530 (359h27m38.872675013s) [noise=true , address=45.91.116.6:4975 , piece_id=SYTAOBSBAKYWFOXNQWVUMB3UPS3VFIYMUX54C4UBWDKZE3TFAHZQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.411076881 (359h27m38.873169354s) [noise=true , address=45.91.116.12:4225 , piece_id=D27PLH63BME3B7SXB2JSIFXDZBU5SULATZH73LPSWKICVQQ3S43A, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.423225625 (359h27m38.885318098s) [noise=true , address=45.91.116.12:4226 , piece_id=4VAARCNHICUT6M7MNE2CLDOE7COG3U5IQ25VNZOPHOERWZ4BXQWQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.454938643 (359h27m38.917031126s) [noise=true , address=45.91.116.7:4009 , piece_id=FM5TRV2QHP6N2NSBN7ZTZEKCRNLCQ7BCIQLNVLGQNCK3P5EH7HEQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.458700767 (359h27m38.92079324s) [noise=true , address=45.91.116.13:4154 , piece_id=NX7N3FKWZWV4HN2I77CL4DHNJ6ZPXAP2FAODVTUO2552ZEF5B43A, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.478635286 (359h27m38.940727759s) [noise=true , address=45.91.116.4:4939 , piece_id=MWKWCCIKHUMKUJGPDE7WGJSIVHV3HAP6PE4SSGCRQ5QYR6VUFXOQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.488485903 (359h27m38.950578386s) [noise=true , address=45.91.116.14:4191 , piece_id=PVMS6GNKYMWPJFHS7QSO2HE3YXNZDSB65F3P4XEXV4VYWTFDPQEQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: <nil> Oct 8 12:58:53.488505601 (359h27m38.950598084s) [seg_pos={0 0}, upload=316262]: Segment reached optimal threshold of 13 pieces. Oct 8 12:58:53.488615380 (359h27m38.950707893s) [noise=true , address=45.91.116.6:4974 , piece_id=JQYTQTT5BSHCVIV563BKRQ6RKJCSZWFH5573CKW7BMZKBPGEHHPQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 8 12:58:53.488629176 (359h27m38.950721689s) [noise=true , address=45.91.116.6:4974 , piece_id=JQYTQTT5BSHCVIV563BKRQ6RKJCSZWFH5573CKW7BMZKBPGEHHPQ, seg_pos={0 0}, upload=316262]: Long tail cancellation due to optimal threshold Oct 8 12:58:53.488761809 (359h27m38.950854292s) [noise=true , address=77.90.31.13:4626 , piece_id=CY5OQBVPCQE7GON6XDUBA5WAM4KAA3ZEYBO4SXZ4ATQ5C74JR6OA, seg_pos={0 0}, upload=316262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Oct 8 12:58:53.488769443 (359h27m38.950861916s) [noise=true , address=77.90.31.13:4626 , piece_id=CY5OQBVPCQE7GON6XDUBA5WAM4KAA3ZEYBO4SXZ4ATQ5C74JR6OA, seg_pos={0 0}, upload=316262]: Long tail cancellation due to optimal threshold Oct 8 12:58:53.488839506 (359h27m38.950931989s) [noise=true , address=45.91.116.12:4227 , piece_id=5RC6EVOMUKDJLHAE7BGIVT7EIN23P6GJVRVYFHYLQPLW5KDP3SZQ, seg_pos={0 0}, upload=316262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Oct 8 12:58:53.488845268 (359h27m38.950937751s) [noise=true , address=45.91.116.12:4227 , piece_id=5RC6EVOMUKDJLHAE7BGIVT7EIN23P6GJVRVYFHYLQPLW5KDP3SZQ, seg_pos={0 0}, upload=316262]: Long tail cancellation due to optimal threshold Oct 8 12:58:53.488861008 (359h27m38.950953481s) [seg_pos={0 0}, upload=316262]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:58:53.488923888 (359h27m38.951016361s) [upload=316262]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:58:53.488928937 (359h27m38.95102142s) [upload=316262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:58:53.875015854 (359h27m39.337108337s) [upload=316262]: Done uploading segments... Oct 8 12:58:53.875040161 (359h27m39.337132644s) [upload=316262]: Upload finished. err: <nil> Duration: 30.001926365s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.347077003s FAIL (2025-10-08 12:58:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.347109294s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [........X........X..X..X....XX..XXX.XX..X.XX..X........XX..X] duration: 30.002022098s FAIL (2025-10-08 12:58:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: DW5VEZD3K6CA3BOEJHRISJXRR6YYLPPXLVA2G4M3UPTBEP2QCWWA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: UYO7RFW4X3W63Z3QHVHILH62LI2IO4YMYX4IKE5ZZW76CBIDABQQ) context deadline exceeded ======LOGS====== 12:57:40.021048226 (359h26m25.483140699s) [upload=316245]: Starting upload Oct 8 12:57:40.021121034 (359h26m25.483213517s) [upload=316245]: Done starting upload Oct 8 12:57:40.021218721 (359h26m25.483311204s) [upload=316245]: Uploading segments... Oct 8 12:57:40.021357244 (359h26m25.483449717s) [upload=316245]: Got next segment. Inline: false Oct 8 12:57:40.021402791 (359h26m25.483495274s) [upload=316245]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:57:40.021410156 (359h26m25.483502629s) [upload=316245]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:57:40.270251810 (359h26m25.732344293s) [seg_pos={0 0}, upload=316245]: Begin upload segment... Oct 8 12:57:40.270299650 (359h26m25.732392123s) [seg_pos={0 0}, upload=316245]: Done begin upload segment. Oct 8 12:57:40.270317976 (359h26m25.732410449s) [noise=true , address=77.90.31.19:4841 , piece_id=UYO7RFW4X3W63Z3QHVHILH62LI2IO4YMYX4IKE5ZZW76CBIDABQQ, seg_pos={0 0}, upload=316245]: Uploading piece... Oct 8 12:57:40.270332403 (359h26m25.732424876s) [noise=true , address=77.90.31.12:4589 , piece_id=OKWYUF2TYJ734A3V4DWH23GH5JDMMKT5IFELKJOT2QL5N7ALLLUQ, seg_pos={0 0}, upload=316245]: Uploading piece... Oct 8 12:57:40.270991939 (359h26m25.733084422s) [noise=true , address=145.79.227.6:4257 , piece_id=CTQYV67VNCIJUEWZJTJWP32LSKXLUJDW5SQTLJCABPZOBWQECNWQ, seg_pos={0 0}, upload=316245]: Uploading piece... Oct 8 12:57:40.271333860 (359h26m25.733426333s) [noise=true , address=145.79.227.6:4259 , piece_id=QCBTMKXPHVYPHYKC6PB6YBZJHXSKAB23NVHNAMJYDSVDVIQOBGAQ, seg_pos={0 0}, upload=316245]: Uploading piece... Oct 8 12:57:40.271597932 (359h26m25.733690415s) [noise=true , address=145.79.227.17:4970 , piece_id=IXNN33CAZIW5JSLCC7K3DEJQ7HTQTZ2YQ3ZZK22NP773BKAVYNNQ, seg_pos={0 0}, upload=316245]: Uploading piece... Oct 8 12:57:40.271848710 (359h26m25.733941193s) [noise=true , address=45.91.116.13:4155 , piece_id=B4XMUJBB3YQFU2OL2LT2FI3OCA5ZSBTT2QVDL22KGJVFWSN62E5A, seg_pos={0 0}, upload=316245]: Uploading piece... Oct 8 12:57:40.271855283 (359h26m25.733947766s) [noise=true , address=45.91.116.8:4047 , piece_id=CBWCH2VJHKHMXHBYYNXUQBPNEUHVB3Y4KR22REKNFBIPC6LDVVMA, seg_pos={0 0}, upload=316245]: Uploading piece... Oct 8 12:57:40.272162278 (359h26m25.734254751s) [noise=true , address=145.79.227.3:4151 , piece_id=24HU54GNWPVUCUTBXSLHF6FL4NNWWXCZH6EDQDW63EIJBDZCM44Q, seg_pos={0 0}, upload=316245]: Uploading piece... Oct 8 12:57:40.272170984 (359h26m25.734263457s) [noise=true , address=45.91.116.10:4081 , piece_id=AQ4DCKNEQNC5JHB67VPAFZCQV3TZNNN7G765ZM2OPHCCIFV77MRA, seg_pos={0 0}, upload=316245]: Uploading piece... Oct 8 12:57:40.272586595 (359h26m25.734679079s) [noise=true , address=45.91.116.5:4298 , piece_id=AMLD3ELIIJEVNDUXM2CTP3HGORACRUXWOJ7C4XGWMOQZUIFDUMOA, seg_pos={0 0}, upload=316245]: Uploading piece... Oct 8 12:57:40.272860948 (359h26m25.734953421s) [noise=true , address=145.79.227.7:4295 , piece_id=MICLBV6SCY4TGNBMCYLYH6TQYMTJMTJDKG67RGNR4Y5XVUA65P3Q, seg_pos={0 0}, upload=316245]: Uploading piece... Oct 8 12:57:40.273263444 (359h26m25.735355917s) [noise=true , address=45.91.116.9:4262 , piece_id=J67UAFE47QV4SRB4XW4KYSM6JN5BD7WM7IDGR24JM3WY3HJCYBMQ, seg_pos={0 0}, upload=316245]: Uploading piece... Oct 8 12:57:40.273458415 (359h26m25.735550889s) [noise=true , address=45.91.116.8:4045 , piece_id=DW5VEZD3K6CA3BOEJHRISJXRR6YYLPPXLVA2G4M3UPTBEP2QCWWA, seg_pos={0 0}, upload=316245]: Uploading piece... Oct 8 12:57:40.273768977 (359h26m25.73586145s) [noise=true , address=45.91.116.14:4190 , piece_id=4K2R53KNN3ODVELFUOJDSDPVPQNACL7YHOUMDNCWOUF62XBIRXKA, seg_pos={0 0}, upload=316245]: Uploading piece... Oct 8 12:57:40.274127209 (359h26m25.736219682s) [noise=true , address=45.91.116.6:4973 , piece_id=LLW5BVYN7ZDXQZJ46CH7WCYXR64QHPZ57BQ7MOJGWYSRJTOGVTIA, seg_pos={0 0}, upload=316245]: Uploading piece... Oct 8 12:57:40.274184348 (359h26m25.736276821s) [noise=true , address=145.79.227.3:4150 , piece_id=32YIMOVDSK4ZEEEINSPGH3GFT26T2WMO5S3LDJ3R4BSKCAK67Y4Q, seg_pos={0 0}, upload=316245]: Uploading piece... Oct 8 12:57:41.974610404 (359h26m27.436702877s) [upload=316245]: Next returned nil segment Oct 8 12:57:41.974632345 (359h26m27.436724828s) [upload=316245]: Waiting for error group managing segments... Oct 8 12:57:42.254614958 (359h26m27.716707441s) [noise=true , address=145.79.227.6:4259 , piece_id=QCBTMKXPHVYPHYKC6PB6YBZJHXSKAB23NVHNAMJYDSVDVIQOBGAQ, seg_pos={0 0}, upload=316245]: Done uploading piece. err: <nil> Oct 8 12:57:42.266407625 (359h26m27.728500108s) [noise=true , address=145.79.227.17:4970 , piece_id=IXNN33CAZIW5JSLCC7K3DEJQ7HTQTZ2YQ3ZZK22NP773BKAVYNNQ, seg_pos={0 0}, upload=316245]: Done uploading piece. err: <nil> Oct 8 12:57:42.267881391 (359h26m27.729973864s) [noise=true , address=145.79.227.7:4295 , piece_id=MICLBV6SCY4TGNBMCYLYH6TQYMTJMTJDKG67RGNR4Y5XVUA65P3Q, seg_pos={0 0}, upload=316245]: Done uploading piece. err: <nil> Oct 8 12:57:42.272982414 (359h26m27.735074887s) [noise=true , address=145.79.227.3:4151 , piece_id=24HU54GNWPVUCUTBXSLHF6FL4NNWWXCZH6EDQDW63EIJBDZCM44Q, seg_pos={0 0}, upload=316245]: Done uploading piece. err: <nil> Oct 8 12:57:42.273373309 (359h26m27.735465782s) [noise=true , address=145.79.227.6:4257 , piece_id=CTQYV67VNCIJUEWZJTJWP32LSKXLUJDW5SQTLJCABPZOBWQECNWQ, seg_pos={0 0}, upload=316245]: Done uploading piece. err: <nil> Oct 8 12:57:42.287801934 (359h26m27.749894407s) [noise=true , address=145.79.227.3:4150 , piece_id=32YIMOVDSK4ZEEEINSPGH3GFT26T2WMO5S3LDJ3R4BSKCAK67Y4Q, seg_pos={0 0}, upload=316245]: Done uploading piece. err: <nil> Oct 8 12:57:42.538395394 (359h26m28.000487877s) [noise=true , address=45.91.116.6:4973 , piece_id=LLW5BVYN7ZDXQZJ46CH7WCYXR64QHPZ57BQ7MOJGWYSRJTOGVTIA, seg_pos={0 0}, upload=316245]: Done uploading piece. err: <nil> Oct 8 12:57:42.566118665 (359h26m28.028211138s) [noise=true , address=77.90.31.12:4589 , piece_id=OKWYUF2TYJ734A3V4DWH23GH5JDMMKT5IFELKJOT2QL5N7ALLLUQ, seg_pos={0 0}, upload=316245]: Done uploading piece. err: <nil> Oct 8 12:57:42.566380223 (359h26m28.028472696s) [noise=true , address=77.90.31.19:4841 , piece_id=UYO7RFW4X3W63Z3QHVHILH62LI2IO4YMYX4IKE5ZZW76CBIDABQQ, seg_pos={0 0}, upload=316245]: Done uploading piece. err: <nil> Oct 8 12:57:42.590552101 (359h26m28.052644584s) [noise=true , address=45.91.116.9:4262 , piece_id=J67UAFE47QV4SRB4XW4KYSM6JN5BD7WM7IDGR24JM3WY3HJCYBMQ, seg_pos={0 0}, upload=316245]: Done uploading piece. err: <nil> Oct 8 12:57:42.629092069 (359h26m28.091184542s) [noise=true , address=45.91.116.8:4045 , piece_id=DW5VEZD3K6CA3BOEJHRISJXRR6YYLPPXLVA2G4M3UPTBEP2QCWWA, seg_pos={0 0}, upload=316245]: Done uploading piece. err: <nil> Oct 8 12:57:42.635738344 (359h26m28.097830818s) [noise=true , address=45.91.116.13:4155 , piece_id=B4XMUJBB3YQFU2OL2LT2FI3OCA5ZSBTT2QVDL22KGJVFWSN62E5A, seg_pos={0 0}, upload=316245]: Done uploading piece. err: <nil> Oct 8 12:57:42.673214257 (359h26m28.13530673s) [noise=true , address=45.91.116.14:4190 , piece_id=4K2R53KNN3ODVELFUOJDSDPVPQNACL7YHOUMDNCWOUF62XBIRXKA, seg_pos={0 0}, upload=316245]: Done uploading piece. err: <nil> Oct 8 12:57:42.673242821 (359h26m28.135335304s) [seg_pos={0 0}, upload=316245]: Segment reached optimal threshold of 13 pieces. Oct 8 12:57:42.673414618 (359h26m28.135507101s) [noise=true , address=45.91.116.10:4081 , piece_id=AQ4DCKNEQNC5JHB67VPAFZCQV3TZNNN7G765ZM2OPHCCIFV77MRA, seg_pos={0 0}, upload=316245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:40606->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:40606->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40606->45.91.116.10:4081: use of closed network connection Oct 8 12:57:42.673426010 (359h26m28.135518493s) [noise=true , address=45.91.116.10:4081 , piece_id=AQ4DCKNEQNC5JHB67VPAFZCQV3TZNNN7G765ZM2OPHCCIFV77MRA, seg_pos={0 0}, upload=316245]: Long tail cancellation due to optimal threshold Oct 8 12:57:42.673538083 (359h26m28.135630566s) [noise=true , address=45.91.116.5:4298 , piece_id=AMLD3ELIIJEVNDUXM2CTP3HGORACRUXWOJ7C4XGWMOQZUIFDUMOA, seg_pos={0 0}, upload=316245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 8 12:57:42.673547632 (359h26m28.135640115s) [noise=true , address=45.91.116.5:4298 , piece_id=AMLD3ELIIJEVNDUXM2CTP3HGORACRUXWOJ7C4XGWMOQZUIFDUMOA, seg_pos={0 0}, upload=316245]: Long tail cancellation due to optimal threshold Oct 8 12:57:42.673554785 (359h26m28.135647268s) [noise=true , address=45.91.116.8:4047 , piece_id=CBWCH2VJHKHMXHBYYNXUQBPNEUHVB3Y4KR22REKNFBIPC6LDVVMA, seg_pos={0 0}, upload=316245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Oct 8 12:57:42.673561408 (359h26m28.135653891s) [noise=true , address=45.91.116.8:4047 , piece_id=CBWCH2VJHKHMXHBYYNXUQBPNEUHVB3Y4KR22REKNFBIPC6LDVVMA, seg_pos={0 0}, upload=316245]: Long tail cancellation due to optimal threshold Oct 8 12:57:42.673581627 (359h26m28.1356741s) [seg_pos={0 0}, upload=316245]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:57:42.673708287 (359h26m28.13580077s) [upload=316245]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:57:42.673717564 (359h26m28.135810048s) [upload=316245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:57:42.985627724 (359h26m28.447720197s) [upload=316245]: Done uploading segments... Oct 8 12:57:42.985648755 (359h26m28.447741238s) [upload=316245]: Upload finished. err: <nil> Duration: 30.002050962s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.........X........X..X..X....XX..XXX.XX..X.XX..X........XX..] duration: 11.793019524s FAIL (2025-10-08 12:57:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: 6TKAF6S3HY65VESSQVDVRHSVZXM7OP6NUM73O2O65A3Q632IAQMQ) context deadline exceeded; piecestore: (Node ID: 12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, Piece ID: HRG7VBLDXQI5J4G332JPCXTF7Y5FGCTNT2QQ7XM6XHQ6JKDLC2TQ) context deadline exceeded ======LOGS====== Oct 8 12:56:23.316480333 (359h25m8.778572806s) [upload=316227]: Starting upload Oct 8 12:56:23.316567879 (359h25m8.778660353s) [upload=316227]: Done starting upload Oct 8 12:56:23.316668692 (359h25m8.778761175s) [upload=316227]: Uploading segments... Oct 8 12:56:23.316793009 (359h25m8.778885492s) [upload=316227]: Got next segment. Inline: false Oct 8 12:56:23.316844957 (359h25m8.77893743s) [upload=316227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:56:23.316849236 (359h25m8.778941709s) [upload=316227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:56:23.642880174 (359h25m9.104972658s) [seg_pos={0 0}, upload=316227]: Begin upload segment... Oct 8 12:56:23.642944958 (359h25m9.105037431s) [seg_pos={0 0}, upload=316227]: Done begin upload segment. Oct 8 12:56:23.643003911 (359h25m9.105096394s) [noise=true , address=145.79.227.3:4151 , piece_id=EV575ELNFLTCNE2MJ3OJZKDFDUYRNISXRHYCCSJ6NK6FRPEJVJGQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.643018247 (359h25m9.105110731s) [noise=true , address=77.90.31.14:4662 , piece_id=3DEO3INOJIDI75TUR4PG6W5VLM6U7LICLU3THZLE5ESOAZOQ45YA, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.643497921 (359h25m9.105590404s) [noise=true , address=77.90.31.10:4517 , piece_id=6TKAF6S3HY65VESSQVDVRHSVZXM7OP6NUM73O2O65A3Q632IAQMQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.643967735 (359h25m9.106060218s) [noise=true , address=145.79.227.6:4257 , piece_id=V57UCBUFK2ZWPGVHJ3SPMDLHMIETKLVDRETIO6SZN4BGMZMJPMPQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.644505410 (359h25m9.106597893s) [noise=true , address=77.90.31.13:4626 , piece_id=J2627G3H5FDOLPX2PC3I5C5O2LCE7X6SP6Z6HKZUQ56L63PPPD5Q, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.644934566 (359h25m9.107027049s) [noise=true , address=77.90.31.19:4842 , piece_id=L5YD5N5BCCGECV2MMSLOLRNOY26J7LWPXUGYGQVC2SKE3JG6T4KQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.646202781 (359h25m9.108295274s) [noise=true , address=45.91.116.8:4048 , piece_id=IIUNY6NIPOSALDOEQ4FWUBPDHLL2YMAQAVOLXSSRQQAN34W7CZUA, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.647137962 (359h25m9.109230445s) [noise=true , address=77.90.31.10:4518 , piece_id=XPOM4RKSTY7AF32US57PGPJVCA4EX3QRMSPDTKHWZOYIG6B3U3JQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.648196087 (359h25m9.11028857s) [noise=true , address=45.91.116.13:4153 , piece_id=EYVZMUL5E6XTBATLSPBTTUZKKZGCC7PB5UMFZTWT4MY4GDNBMZRQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.649465263 (359h25m9.111557757s) [noise=true , address=77.90.31.15:4699 , piece_id=HRG7VBLDXQI5J4G332JPCXTF7Y5FGCTNT2QQ7XM6XHQ6JKDLC2TQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.649746158 (359h25m9.111838641s) [noise=true , address=45.91.116.10:4082 , piece_id=NAQYDT6Z7ZGYTPTGZBYP2RSRWZOXEAI7DVP7ZECRB2P5BRGXNMFQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.650922829 (359h25m9.113015312s) [noise=true , address=77.90.31.11:4553 , piece_id=F7WH6724D75ADQY5YGFQMV72XOWN6MVKZXZEVFC7CGXAQLMCEQPA, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.651968650 (359h25m9.114061144s) [noise=true , address=45.91.116.7:4009 , piece_id=4L4JY6ADZD5NXLFTNT5UFEAVU7WJC5QBQWVWH4HH5QGAGXE2F44A, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.652055126 (359h25m9.114147609s) [noise=true , address=45.91.116.4:4938 , piece_id=YA56TMIHQAH4EKTJKNUENFJYQZCMJZ6CYA3LX6AY4SYAKNVT6SVA, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.654150736 (359h25m9.116243229s) [noise=true , address=145.79.227.14:4838 , piece_id=X25Z7JCDVW26LONGF5AFEVKL4UWLSHJRBHQMFFH6RVWTU44WGISA, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.656136076 (359h25m9.118228559s) [noise=true , address=45.91.116.13:4155 , piece_id=JK47IENDYXFCOXRM3H4NVLBFYBQEKPKFMQP3VSUF2RE7WS3GDH7A, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:25.363084094 (359h25m10.825176567s) [upload=316227]: Next returned nil segment Oct 8 12:56:25.363106306 (359h25m10.825198789s) [upload=316227]: Waiting for error group managing segments... Oct 8 12:56:25.640618136 (359h25m11.102710619s) [noise=true , address=145.79.227.14:4838 , piece_id=X25Z7JCDVW26LONGF5AFEVKL4UWLSHJRBHQMFFH6RVWTU44WGISA, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:25.653102529 (359h25m11.115195012s) [noise=true , address=145.79.227.6:4257 , piece_id=V57UCBUFK2ZWPGVHJ3SPMDLHMIETKLVDRETIO6SZN4BGMZMJPMPQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:25.672153866 (359h25m11.134246349s) [noise=true , address=145.79.227.3:4151 , piece_id=EV575ELNFLTCNE2MJ3OJZKDFDUYRNISXRHYCCSJ6NK6FRPEJVJGQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:25.828753564 (359h25m11.290846037s) [noise=true , address=45.91.116.8:4048 , piece_id=IIUNY6NIPOSALDOEQ4FWUBPDHLL2YMAQAVOLXSSRQQAN34W7CZUA, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:25.913972929 (359h25m11.376065412s) [noise=true , address=77.90.31.13:4626 , piece_id=J2627G3H5FDOLPX2PC3I5C5O2LCE7X6SP6Z6HKZUQ56L63PPPD5Q, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:25.915212830 (359h25m11.377305303s) [noise=true , address=77.90.31.11:4553 , piece_id=F7WH6724D75ADQY5YGFQMV72XOWN6MVKZXZEVFC7CGXAQLMCEQPA, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:25.999465695 (359h25m11.461558168s) [noise=true , address=77.90.31.10:4517 , piece_id=6TKAF6S3HY65VESSQVDVRHSVZXM7OP6NUM73O2O65A3Q632IAQMQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.020706001 (359h25m11.482798484s) [noise=true , address=77.90.31.19:4842 , piece_id=L5YD5N5BCCGECV2MMSLOLRNOY26J7LWPXUGYGQVC2SKE3JG6T4KQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.031584084 (359h25m11.493676567s) [noise=true , address=77.90.31.15:4699 , piece_id=HRG7VBLDXQI5J4G332JPCXTF7Y5FGCTNT2QQ7XM6XHQ6JKDLC2TQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.055310114 (359h25m11.517402587s) [noise=true , address=45.91.116.10:4082 , piece_id=NAQYDT6Z7ZGYTPTGZBYP2RSRWZOXEAI7DVP7ZECRB2P5BRGXNMFQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.099570755 (359h25m11.561663228s) [noise=true , address=45.91.116.13:4153 , piece_id=EYVZMUL5E6XTBATLSPBTTUZKKZGCC7PB5UMFZTWT4MY4GDNBMZRQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.116957122 (359h25m11.579049605s) [noise=true , address=77.90.31.14:4662 , piece_id=3DEO3INOJIDI75TUR4PG6W5VLM6U7LICLU3THZLE5ESOAZOQ45YA, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.385372890 (359h25m11.847465373s) [noise=true , address=45.91.116.13:4155 , piece_id=JK47IENDYXFCOXRM3H4NVLBFYBQEKPKFMQP3VSUF2RE7WS3GDH7A, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.385412625 (359h25m11.847505099s) [seg_pos={0 0}, upload=316227]: Segment reached optimal threshold of 13 pieces. Oct 8 12:56:26.385539276 (359h25m11.847631759s) [noise=true , address=77.90.31.10:4518 , piece_id=XPOM4RKSTY7AF32US57PGPJVCA4EX3QRMSPDTKHWZOYIG6B3U3JQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Oct 8 12:56:26.385548554 (359h25m11.847641037s) [noise=true , address=77.90.31.10:4518 , piece_id=XPOM4RKSTY7AF32US57PGPJVCA4EX3QRMSPDTKHWZOYIG6B3U3JQ, seg_pos={0 0}, upload=316227]: Long tail cancellation due to optimal threshold Oct 8 12:56:26.385593229 (359h25m11.847685702s) [noise=true , address=45.91.116.7:4009 , piece_id=4L4JY6ADZD5NXLFTNT5UFEAVU7WJC5QBQWVWH4HH5QGAGXE2F44A, seg_pos={0 0}, upload=316227]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Oct 8 12:56:26.385602837 (359h25m11.84769532s) [noise=true , address=45.91.116.7:4009 , piece_id=4L4JY6ADZD5NXLFTNT5UFEAVU7WJC5QBQWVWH4HH5QGAGXE2F44A, seg_pos={0 0}, upload=316227]: Long tail cancellation due to optimal threshold Oct 8 12:56:26.385654115 (359h25m11.847746598s) [noise=true , address=45.91.116.4:4938 , piece_id=YA56TMIHQAH4EKTJKNUENFJYQZCMJZ6CYA3LX6AY4SYAKNVT6SVA, seg_pos={0 0}, upload=316227]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Oct 8 12:56:26.385663653 (359h25m11.847756136s) [noise=true , address=45.91.116.4:4938 , piece_id=YA56TMIHQAH4EKTJKNUENFJYQZCMJZ6CYA3LX6AY4SYAKNVT6SVA, seg_pos={0 0}, upload=316227]: Long tail cancellation due to optimal threshold Oct 8 12:56:26.385702066 (359h25m11.847794549s) [seg_pos={0 0}, upload=316227]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:56:26.385832375 (359h25m11.847924868s) [upload=316227]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:56:26.385841472 (359h25m11.847933945s) [upload=316227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:56:26.709442534 (359h25m12.171535037s) [upload=316227]: Done uploading segments... Oct 8 12:56:26.709469434 (359h25m12.171561948s) [upload=316227]: Upload finished. err: <nil> Duration: 11.793054431s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.214638792s FAIL (2025-10-08 12:57:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.21465876s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..........X........X..X..X....XX..XXX.XX..X.XX..X........XX.] duration: 6.120968714s FAIL (2025-10-08 12:57:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: 6TKAF6S3HY65VESSQVDVRHSVZXM7OP6NUM73O2O65A3Q632IAQMQ) context deadline exceeded; piecestore: (Node ID: 12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, Piece ID: HRG7VBLDXQI5J4G332JPCXTF7Y5FGCTNT2QQ7XM6XHQ6JKDLC2TQ) context deadline exceeded ======LOGS====== Oct 8 12:56:23.316480333 (359h25m8.778572806s) [upload=316227]: Starting upload Oct 8 12:56:23.316567879 (359h25m8.778660353s) [upload=316227]: Done starting upload Oct 8 12:56:23.316668692 (359h25m8.778761175s) [upload=316227]: Uploading segments... Oct 8 12:56:23.316793009 (359h25m8.778885492s) [upload=316227]: Got next segment. Inline: false Oct 8 12:56:23.316844957 (359h25m8.77893743s) [upload=316227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:56:23.316849236 (359h25m8.778941709s) [upload=316227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:56:23.642880174 (359h25m9.104972658s) [seg_pos={0 0}, upload=316227]: Begin upload segment... Oct 8 12:56:23.642944958 (359h25m9.105037431s) [seg_pos={0 0}, upload=316227]: Done begin upload segment. Oct 8 12:56:23.643003911 (359h25m9.105096394s) [noise=true , address=145.79.227.3:4151 , piece_id=EV575ELNFLTCNE2MJ3OJZKDFDUYRNISXRHYCCSJ6NK6FRPEJVJGQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.643018247 (359h25m9.105110731s) [noise=true , address=77.90.31.14:4662 , piece_id=3DEO3INOJIDI75TUR4PG6W5VLM6U7LICLU3THZLE5ESOAZOQ45YA, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.643497921 (359h25m9.105590404s) [noise=true , address=77.90.31.10:4517 , piece_id=6TKAF6S3HY65VESSQVDVRHSVZXM7OP6NUM73O2O65A3Q632IAQMQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.643967735 (359h25m9.106060218s) [noise=true , address=145.79.227.6:4257 , piece_id=V57UCBUFK2ZWPGVHJ3SPMDLHMIETKLVDRETIO6SZN4BGMZMJPMPQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.644505410 (359h25m9.106597893s) [noise=true , address=77.90.31.13:4626 , piece_id=J2627G3H5FDOLPX2PC3I5C5O2LCE7X6SP6Z6HKZUQ56L63PPPD5Q, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.644934566 (359h25m9.107027049s) [noise=true , address=77.90.31.19:4842 , piece_id=L5YD5N5BCCGECV2MMSLOLRNOY26J7LWPXUGYGQVC2SKE3JG6T4KQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.646202781 (359h25m9.108295274s) [noise=true , address=45.91.116.8:4048 , piece_id=IIUNY6NIPOSALDOEQ4FWUBPDHLL2YMAQAVOLXSSRQQAN34W7CZUA, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.647137962 (359h25m9.109230445s) [noise=true , address=77.90.31.10:4518 , piece_id=XPOM4RKSTY7AF32US57PGPJVCA4EX3QRMSPDTKHWZOYIG6B3U3JQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.648196087 (359h25m9.11028857s) [noise=true , address=45.91.116.13:4153 , piece_id=EYVZMUL5E6XTBATLSPBTTUZKKZGCC7PB5UMFZTWT4MY4GDNBMZRQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.649465263 (359h25m9.111557757s) [noise=true , address=77.90.31.15:4699 , piece_id=HRG7VBLDXQI5J4G332JPCXTF7Y5FGCTNT2QQ7XM6XHQ6JKDLC2TQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.649746158 (359h25m9.111838641s) [noise=true , address=45.91.116.10:4082 , piece_id=NAQYDT6Z7ZGYTPTGZBYP2RSRWZOXEAI7DVP7ZECRB2P5BRGXNMFQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.650922829 (359h25m9.113015312s) [noise=true , address=77.90.31.11:4553 , piece_id=F7WH6724D75ADQY5YGFQMV72XOWN6MVKZXZEVFC7CGXAQLMCEQPA, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.651968650 (359h25m9.114061144s) [noise=true , address=45.91.116.7:4009 , piece_id=4L4JY6ADZD5NXLFTNT5UFEAVU7WJC5QBQWVWH4HH5QGAGXE2F44A, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.652055126 (359h25m9.114147609s) [noise=true , address=45.91.116.4:4938 , piece_id=YA56TMIHQAH4EKTJKNUENFJYQZCMJZ6CYA3LX6AY4SYAKNVT6SVA, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.654150736 (359h25m9.116243229s) [noise=true , address=145.79.227.14:4838 , piece_id=X25Z7JCDVW26LONGF5AFEVKL4UWLSHJRBHQMFFH6RVWTU44WGISA, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.656136076 (359h25m9.118228559s) [noise=true , address=45.91.116.13:4155 , piece_id=JK47IENDYXFCOXRM3H4NVLBFYBQEKPKFMQP3VSUF2RE7WS3GDH7A, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:25.363084094 (359h25m10.825176567s) [upload=316227]: Next returned nil segment Oct 8 12:56:25.363106306 (359h25m10.825198789s) [upload=316227]: Waiting for error group managing segments... Oct 8 12:56:25.640618136 (359h25m11.102710619s) [noise=true , address=145.79.227.14:4838 , piece_id=X25Z7JCDVW26LONGF5AFEVKL4UWLSHJRBHQMFFH6RVWTU44WGISA, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:25.653102529 (359h25m11.115195012s) [noise=true , address=145.79.227.6:4257 , piece_id=V57UCBUFK2ZWPGVHJ3SPMDLHMIETKLVDRETIO6SZN4BGMZMJPMPQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:25.672153866 (359h25m11.134246349s) [noise=true , address=145.79.227.3:4151 , piece_id=EV575ELNFLTCNE2MJ3OJZKDFDUYRNISXRHYCCSJ6NK6FRPEJVJGQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:25.828753564 (359h25m11.290846037s) [noise=true , address=45.91.116.8:4048 , piece_id=IIUNY6NIPOSALDOEQ4FWUBPDHLL2YMAQAVOLXSSRQQAN34W7CZUA, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:25.913972929 (359h25m11.376065412s) [noise=true , address=77.90.31.13:4626 , piece_id=J2627G3H5FDOLPX2PC3I5C5O2LCE7X6SP6Z6HKZUQ56L63PPPD5Q, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:25.915212830 (359h25m11.377305303s) [noise=true , address=77.90.31.11:4553 , piece_id=F7WH6724D75ADQY5YGFQMV72XOWN6MVKZXZEVFC7CGXAQLMCEQPA, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:25.999465695 (359h25m11.461558168s) [noise=true , address=77.90.31.10:4517 , piece_id=6TKAF6S3HY65VESSQVDVRHSVZXM7OP6NUM73O2O65A3Q632IAQMQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.020706001 (359h25m11.482798484s) [noise=true , address=77.90.31.19:4842 , piece_id=L5YD5N5BCCGECV2MMSLOLRNOY26J7LWPXUGYGQVC2SKE3JG6T4KQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.031584084 (359h25m11.493676567s) [noise=true , address=77.90.31.15:4699 , piece_id=HRG7VBLDXQI5J4G332JPCXTF7Y5FGCTNT2QQ7XM6XHQ6JKDLC2TQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.055310114 (359h25m11.517402587s) [noise=true , address=45.91.116.10:4082 , piece_id=NAQYDT6Z7ZGYTPTGZBYP2RSRWZOXEAI7DVP7ZECRB2P5BRGXNMFQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.099570755 (359h25m11.561663228s) [noise=true , address=45.91.116.13:4153 , piece_id=EYVZMUL5E6XTBATLSPBTTUZKKZGCC7PB5UMFZTWT4MY4GDNBMZRQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.116957122 (359h25m11.579049605s) [noise=true , address=77.90.31.14:4662 , piece_id=3DEO3INOJIDI75TUR4PG6W5VLM6U7LICLU3THZLE5ESOAZOQ45YA, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.385372890 (359h25m11.847465373s) [noise=true , address=45.91.116.13:4155 , piece_id=JK47IENDYXFCOXRM3H4NVLBFYBQEKPKFMQP3VSUF2RE7WS3GDH7A, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.385412625 (359h25m11.847505099s) [seg_pos={0 0}, upload=316227]: Segment reached optimal threshold of 13 pieces. Oct 8 12:56:26.385539276 (359h25m11.847631759s) [noise=true , address=77.90.31.10:4518 , piece_id=XPOM4RKSTY7AF32US57PGPJVCA4EX3QRMSPDTKHWZOYIG6B3U3JQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Oct 8 12:56:26.385548554 (359h25m11.847641037s) [noise=true , address=77.90.31.10:4518 , piece_id=XPOM4RKSTY7AF32US57PGPJVCA4EX3QRMSPDTKHWZOYIG6B3U3JQ, seg_pos={0 0}, upload=316227]: Long tail cancellation due to optimal threshold Oct 8 12:56:26.385593229 (359h25m11.847685702s) [noise=true , address=45.91.116.7:4009 , piece_id=4L4JY6ADZD5NXLFTNT5UFEAVU7WJC5QBQWVWH4HH5QGAGXE2F44A, seg_pos={0 0}, upload=316227]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Oct 8 12:56:26.385602837 (359h25m11.84769532s) [noise=true , address=45.91.116.7:4009 , piece_id=4L4JY6ADZD5NXLFTNT5UFEAVU7WJC5QBQWVWH4HH5QGAGXE2F44A, seg_pos={0 0}, upload=316227]: Long tail cancellation due to optimal threshold Oct 8 12:56:26.385654115 (359h25m11.847746598s) [noise=true , address=45.91.116.4:4938 , piece_id=YA56TMIHQAH4EKTJKNUENFJYQZCMJZ6CYA3LX6AY4SYAKNVT6SVA, seg_pos={0 0}, upload=316227]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Oct 8 12:56:26.385663653 (359h25m11.847756136s) [noise=true , address=45.91.116.4:4938 , piece_id=YA56TMIHQAH4EKTJKNUENFJYQZCMJZ6CYA3LX6AY4SYAKNVT6SVA, seg_pos={0 0}, upload=316227]: Long tail cancellation due to optimal threshold Oct 8 12:56:26.385702066 (359h25m11.847794549s) [seg_pos={0 0}, upload=316227]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:56:26.385832375 (359h25m11.847924868s) [upload=316227]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:56:26.385841472 (359h25m11.847933945s) [upload=316227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:56:26.709442534 (359h25m12.171535037s) [upload=316227]: Done uploading segments... Oct 8 12:56:26.709469434 (359h25m12.171561948s) [upload=316227]: Upload finished. err: <nil> Duration: 6.120997428s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...........X........X..X..X....XX..XXX.XX..X.XX..X........XX] duration: 30.002751863s FAIL (2025-10-08 12:56:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: 6TKAF6S3HY65VESSQVDVRHSVZXM7OP6NUM73O2O65A3Q632IAQMQ) context deadline exceeded; piecestore: (Node ID: 12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, Piece ID: HRG7VBLDXQI5J4G332JPCXTF7Y5FGCTNT2QQ7XM6XHQ6JKDLC2TQ) context deadline exceeded ======LOGS====== Oct 8 12:56:23.316480333 (359h25m8.778572806s) [upload=316227]: Starting upload Oct 8 12:56:23.316567879 (359h25m8.778660353s) [upload=316227]: Done starting upload Oct 8 12:56:23.316668692 (359h25m8.778761175s) [upload=316227]: Uploading segments... Oct 8 12:56:23.316793009 (359h25m8.778885492s) [upload=316227]: Got next segment. Inline: false Oct 8 12:56:23.316844957 (359h25m8.77893743s) [upload=316227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:56:23.316849236 (359h25m8.778941709s) [upload=316227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:56:23.642880174 (359h25m9.104972658s) [seg_pos={0 0}, upload=316227]: Begin upload segment... Oct 8 12:56:23.642944958 (359h25m9.105037431s) [seg_pos={0 0}, upload=316227]: Done begin upload segment. Oct 8 12:56:23.643003911 (359h25m9.105096394s) [noise=true , address=145.79.227.3:4151 , piece_id=EV575ELNFLTCNE2MJ3OJZKDFDUYRNISXRHYCCSJ6NK6FRPEJVJGQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.643018247 (359h25m9.105110731s) [noise=true , address=77.90.31.14:4662 , piece_id=3DEO3INOJIDI75TUR4PG6W5VLM6U7LICLU3THZLE5ESOAZOQ45YA, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.643497921 (359h25m9.105590404s) [noise=true , address=77.90.31.10:4517 , piece_id=6TKAF6S3HY65VESSQVDVRHSVZXM7OP6NUM73O2O65A3Q632IAQMQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.643967735 (359h25m9.106060218s) [noise=true , address=145.79.227.6:4257 , piece_id=V57UCBUFK2ZWPGVHJ3SPMDLHMIETKLVDRETIO6SZN4BGMZMJPMPQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.644505410 (359h25m9.106597893s) [noise=true , address=77.90.31.13:4626 , piece_id=J2627G3H5FDOLPX2PC3I5C5O2LCE7X6SP6Z6HKZUQ56L63PPPD5Q, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.644934566 (359h25m9.107027049s) [noise=true , address=77.90.31.19:4842 , piece_id=L5YD5N5BCCGECV2MMSLOLRNOY26J7LWPXUGYGQVC2SKE3JG6T4KQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.646202781 (359h25m9.108295274s) [noise=true , address=45.91.116.8:4048 , piece_id=IIUNY6NIPOSALDOEQ4FWUBPDHLL2YMAQAVOLXSSRQQAN34W7CZUA, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.647137962 (359h25m9.109230445s) [noise=true , address=77.90.31.10:4518 , piece_id=XPOM4RKSTY7AF32US57PGPJVCA4EX3QRMSPDTKHWZOYIG6B3U3JQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.648196087 (359h25m9.11028857s) [noise=true , address=45.91.116.13:4153 , piece_id=EYVZMUL5E6XTBATLSPBTTUZKKZGCC7PB5UMFZTWT4MY4GDNBMZRQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.649465263 (359h25m9.111557757s) [noise=true , address=77.90.31.15:4699 , piece_id=HRG7VBLDXQI5J4G332JPCXTF7Y5FGCTNT2QQ7XM6XHQ6JKDLC2TQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.649746158 (359h25m9.111838641s) [noise=true , address=45.91.116.10:4082 , piece_id=NAQYDT6Z7ZGYTPTGZBYP2RSRWZOXEAI7DVP7ZECRB2P5BRGXNMFQ, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.650922829 (359h25m9.113015312s) [noise=true , address=77.90.31.11:4553 , piece_id=F7WH6724D75ADQY5YGFQMV72XOWN6MVKZXZEVFC7CGXAQLMCEQPA, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.651968650 (359h25m9.114061144s) [noise=true , address=45.91.116.7:4009 , piece_id=4L4JY6ADZD5NXLFTNT5UFEAVU7WJC5QBQWVWH4HH5QGAGXE2F44A, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.652055126 (359h25m9.114147609s) [noise=true , address=45.91.116.4:4938 , piece_id=YA56TMIHQAH4EKTJKNUENFJYQZCMJZ6CYA3LX6AY4SYAKNVT6SVA, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.654150736 (359h25m9.116243229s) [noise=true , address=145.79.227.14:4838 , piece_id=X25Z7JCDVW26LONGF5AFEVKL4UWLSHJRBHQMFFH6RVWTU44WGISA, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:23.656136076 (359h25m9.118228559s) [noise=true , address=45.91.116.13:4155 , piece_id=JK47IENDYXFCOXRM3H4NVLBFYBQEKPKFMQP3VSUF2RE7WS3GDH7A, seg_pos={0 0}, upload=316227]: Uploading piece... Oct 8 12:56:25.363084094 (359h25m10.825176567s) [upload=316227]: Next returned nil segment Oct 8 12:56:25.363106306 (359h25m10.825198789s) [upload=316227]: Waiting for error group managing segments... Oct 8 12:56:25.640618136 (359h25m11.102710619s) [noise=true , address=145.79.227.14:4838 , piece_id=X25Z7JCDVW26LONGF5AFEVKL4UWLSHJRBHQMFFH6RVWTU44WGISA, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:25.653102529 (359h25m11.115195012s) [noise=true , address=145.79.227.6:4257 , piece_id=V57UCBUFK2ZWPGVHJ3SPMDLHMIETKLVDRETIO6SZN4BGMZMJPMPQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:25.672153866 (359h25m11.134246349s) [noise=true , address=145.79.227.3:4151 , piece_id=EV575ELNFLTCNE2MJ3OJZKDFDUYRNISXRHYCCSJ6NK6FRPEJVJGQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:25.828753564 (359h25m11.290846037s) [noise=true , address=45.91.116.8:4048 , piece_id=IIUNY6NIPOSALDOEQ4FWUBPDHLL2YMAQAVOLXSSRQQAN34W7CZUA, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:25.913972929 (359h25m11.376065412s) [noise=true , address=77.90.31.13:4626 , piece_id=J2627G3H5FDOLPX2PC3I5C5O2LCE7X6SP6Z6HKZUQ56L63PPPD5Q, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:25.915212830 (359h25m11.377305303s) [noise=true , address=77.90.31.11:4553 , piece_id=F7WH6724D75ADQY5YGFQMV72XOWN6MVKZXZEVFC7CGXAQLMCEQPA, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:25.999465695 (359h25m11.461558168s) [noise=true , address=77.90.31.10:4517 , piece_id=6TKAF6S3HY65VESSQVDVRHSVZXM7OP6NUM73O2O65A3Q632IAQMQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.020706001 (359h25m11.482798484s) [noise=true , address=77.90.31.19:4842 , piece_id=L5YD5N5BCCGECV2MMSLOLRNOY26J7LWPXUGYGQVC2SKE3JG6T4KQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.031584084 (359h25m11.493676567s) [noise=true , address=77.90.31.15:4699 , piece_id=HRG7VBLDXQI5J4G332JPCXTF7Y5FGCTNT2QQ7XM6XHQ6JKDLC2TQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.055310114 (359h25m11.517402587s) [noise=true , address=45.91.116.10:4082 , piece_id=NAQYDT6Z7ZGYTPTGZBYP2RSRWZOXEAI7DVP7ZECRB2P5BRGXNMFQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.099570755 (359h25m11.561663228s) [noise=true , address=45.91.116.13:4153 , piece_id=EYVZMUL5E6XTBATLSPBTTUZKKZGCC7PB5UMFZTWT4MY4GDNBMZRQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.116957122 (359h25m11.579049605s) [noise=true , address=77.90.31.14:4662 , piece_id=3DEO3INOJIDI75TUR4PG6W5VLM6U7LICLU3THZLE5ESOAZOQ45YA, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.385372890 (359h25m11.847465373s) [noise=true , address=45.91.116.13:4155 , piece_id=JK47IENDYXFCOXRM3H4NVLBFYBQEKPKFMQP3VSUF2RE7WS3GDH7A, seg_pos={0 0}, upload=316227]: Done uploading piece. err: <nil> Oct 8 12:56:26.385412625 (359h25m11.847505099s) [seg_pos={0 0}, upload=316227]: Segment reached optimal threshold of 13 pieces. Oct 8 12:56:26.385539276 (359h25m11.847631759s) [noise=true , address=77.90.31.10:4518 , piece_id=XPOM4RKSTY7AF32US57PGPJVCA4EX3QRMSPDTKHWZOYIG6B3U3JQ, seg_pos={0 0}, upload=316227]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Oct 8 12:56:26.385548554 (359h25m11.847641037s) [noise=true , address=77.90.31.10:4518 , piece_id=XPOM4RKSTY7AF32US57PGPJVCA4EX3QRMSPDTKHWZOYIG6B3U3JQ, seg_pos={0 0}, upload=316227]: Long tail cancellation due to optimal threshold Oct 8 12:56:26.385593229 (359h25m11.847685702s) [noise=true , address=45.91.116.7:4009 , piece_id=4L4JY6ADZD5NXLFTNT5UFEAVU7WJC5QBQWVWH4HH5QGAGXE2F44A, seg_pos={0 0}, upload=316227]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Oct 8 12:56:26.385602837 (359h25m11.84769532s) [noise=true , address=45.91.116.7:4009 , piece_id=4L4JY6ADZD5NXLFTNT5UFEAVU7WJC5QBQWVWH4HH5QGAGXE2F44A, seg_pos={0 0}, upload=316227]: Long tail cancellation due to optimal threshold Oct 8 12:56:26.385654115 (359h25m11.847746598s) [noise=true , address=45.91.116.4:4938 , piece_id=YA56TMIHQAH4EKTJKNUENFJYQZCMJZ6CYA3LX6AY4SYAKNVT6SVA, seg_pos={0 0}, upload=316227]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Oct 8 12:56:26.385663653 (359h25m11.847756136s) [noise=true , address=45.91.116.4:4938 , piece_id=YA56TMIHQAH4EKTJKNUENFJYQZCMJZ6CYA3LX6AY4SYAKNVT6SVA, seg_pos={0 0}, upload=316227]: Long tail cancellation due to optimal threshold Oct 8 12:56:26.385702066 (359h25m11.847794549s) [seg_pos={0 0}, upload=316227]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:56:26.385832375 (359h25m11.847924868s) [upload=316227]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:56:26.385841472 (359h25m11.847933945s) [upload=316227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:56:26.709442534 (359h25m12.171535037s) [upload=316227]: Done uploading segments... Oct 8 12:56:26.709469434 (359h25m12.171561948s) [upload=316227]: Upload finished. err: <nil> Duration: 30.002771922s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [............X........X..X..X....XX..XXX.XX..X.XX..X........X] duration: 30.002607729s FAIL (2025-10-08 12:56:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53.114497212 (359h24m38.576589695s) [upload=316220]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:55:53.448572428 (359h24m38.910664902s) [seg_pos={0 0}, upload=316220]: Begin upload segment... Oct 8 12:55:53.448645508 (359h24m38.910737991s) [seg_pos={0 0}, upload=316220]: Done begin upload segment. Oct 8 12:55:53.448675715 (359h24m38.910768188s) [noise=true , address=77.90.31.16:4735 , piece_id=PSERZ5IIOBXK52H6SM6H4GMFXO24KDNNWU5GH5YWUHULGG27H5OQ, seg_pos={0 0}, upload=316220]: Uploading piece... Oct 8 12:55:53.448689641 (359h24m38.910782124s) [noise=true , address=145.79.227.11:4731 , piece_id=3OTEDC45G7FV6NMXZ2IAK7RRS7UAWK4OM2T7KWGOFGLZ4UBAXBOA, seg_pos={0 0}, upload=316220]: Uploading piece... Oct 8 12:55:53.449446773 (359h24m38.911539246s) [noise=true , address=145.79.227.15:4874 , piece_id=AKBVGX2DBAEZXXDNA5HB5QBLOCEUKA4QKUOOO3J445UESULVHOPA, seg_pos={0 0}, upload=316220]: Uploading piece... Oct 8 12:55:53.449521215 (359h24m38.911613688s) [noise=true , address=45.91.116.6:4974 , piece_id=KFWEUE5CUCYXYKES2B5DC7DXCNYZIFUYRH3EETNNIRXSWUMNG2NA, seg_pos={0 0}, upload=316220]: Uploading piece... Oct 8 12:55:53.449860561 (359h24m38.911953044s) [noise=true , address=145.79.227.3:4150 , piece_id=WOFGMBFBG5NGBOA6MFNMLR4UL3SAD5UVOEETXV4LS5NLQC2JEZDA, seg_pos={0 0}, upload=316220]: Uploading piece... Oct 8 12:55:53.450120457 (359h24m38.91221293s) [noise=true , address=145.79.227.14:4837 , piece_id=SCPXVH7RBW4X22OS64MBQC4JFHYZDOORRKL4S7XNAMQADZ3UQXAA, seg_pos={0 0}, upload=316220]: Uploading piece... Oct 8 12:55:53.450362337 (359h24m38.91245482s) [noise=true , address=77.90.31.13:4626 , piece_id=LTGV7G3MJZUFIEYZV3SS3XR2O2G2CUBIDQJ7UKRVKWV6NA7EPODQ, seg_pos={0 0}, upload=316220]: Uploading piece... Oct 8 12:55:53.450627391 (359h24m38.912719865s) [noise=true , address=145.79.227.2:4115 , piece_id=3MZT3PR3UUEKW42XU6UECLX64DTUGL6UXSRKUNAPIRA6X3GNIQDA, seg_pos={0 0}, upload=316220]: Uploading piece... Oct 8 12:55:53.450904599 (359h24m38.912997082s) [noise=true , address=77.90.31.10:4520 , piece_id=TX7H5DQWBX6F5FENMQHDWJDDOSLR4K4XVYZGDVEROECM6ZRJR6AQ, seg_pos={0 0}, upload=316220]: Uploading piece... Oct 8 12:55:53.451145268 (359h24m38.913237741s) [noise=true , address=145.79.227.17:4972 , piece_id=4G3SFKWEPJ7JC6IJTPRO5232LPSTKYQIQMXQB2GILFPZLWAMDOJQ, seg_pos={0 0}, upload=316220]: Uploading piece... Oct 8 12:55:53.451383572 (359h24m38.913476055s) [noise=true , address=145.79.227.4:4185 , piece_id=2YZZIOTFSFG7OTZ3ZE3G74H5C7VUCUFAT26ZY7UYZXUACKVQP4KQ, seg_pos={0 0}, upload=316220]: Uploading piece... Oct 8 12:55:53.451673714 (359h24m38.913766187s) [noise=true , address=45.91.116.9:4262 , piece_id=QCVD5JFC7SBAFMQMBZVUP2NFXUSZEAOV5D6FTJ7SBSSKOTS2Q7FQ, seg_pos={0 0}, upload=316220]: Uploading piece... Oct 8 12:55:53.451924482 (359h24m38.914016965s) [noise=true , address=145.79.227.5:4221 , piece_id=JHMBHZY6BHYKFAIIVJ3HIY54PBBHAQH3DCH5WV42JC4PKFIO2VXA, seg_pos={0 0}, upload=316220]: Uploading piece... Oct 8 12:55:53.452176742 (359h24m38.914269215s) [noise=true , address=145.79.227.7:4295 , piece_id=3ESCMJMPBAMJ73DSM446PEKEQZNXP53L2XZDRNFUBMECSHAFYVDA, seg_pos={0 0}, upload=316220]: Uploading piece... Oct 8 12:55:53.452484408 (359h24m38.914576891s) [noise=true , address=145.79.227.13:4801 , piece_id=365WTSG5MCTHEZNJINWNGNULFFZJ4CBKHU32P5CAOPGU5LLW2QIA, seg_pos={0 0}, upload=316220]: Uploading piece... Oct 8 12:55:53.452730296 (359h24m38.914822779s) [noise=true , address=45.91.116.10:4083 , piece_id=SYVWQOBNBJWU34ZFPBJE3NKKGFXEXYMSVKWNQ3X22I3IFRFE53VQ, seg_pos={0 0}, upload=316220]: Uploading piece... Oct 8 12:55:55.150721636 (359h24m40.612814109s) [upload=316220]: Next returned nil segment Oct 8 12:55:55.150739530 (359h24m40.612832003s) [upload=316220]: Waiting for error group managing segments... Oct 8 12:55:55.430586305 (359h24m40.892678788s) [noise=true , address=145.79.227.11:4731 , piece_id=3OTEDC45G7FV6NMXZ2IAK7RRS7UAWK4OM2T7KWGOFGLZ4UBAXBOA, seg_pos={0 0}, upload=316220]: Done uploading piece. err: <nil> Oct 8 12:55:55.431250319 (359h24m40.893342803s) [noise=true , address=145.79.227.2:4115 , piece_id=3MZT3PR3UUEKW42XU6UECLX64DTUGL6UXSRKUNAPIRA6X3GNIQDA, seg_pos={0 0}, upload=316220]: Done uploading piece. err: <nil> Oct 8 12:55:55.432753121 (359h24m40.894845594s) [noise=true , address=145.79.227.14:4837 , piece_id=SCPXVH7RBW4X22OS64MBQC4JFHYZDOORRKL4S7XNAMQADZ3UQXAA, seg_pos={0 0}, upload=316220]: Done uploading piece. err: <nil> Oct 8 12:55:55.432934787 (359h24m40.89502727s) [noise=true , address=145.79.227.15:4874 , piece_id=AKBVGX2DBAEZXXDNA5HB5QBLOCEUKA4QKUOOO3J445UESULVHOPA, seg_pos={0 0}, upload=316220]: Done uploading piece. err: <nil> Oct 8 12:55:55.441099714 (359h24m40.903192198s) [noise=true , address=145.79.227.4:4185 , piece_id=2YZZIOTFSFG7OTZ3ZE3G74H5C7VUCUFAT26ZY7UYZXUACKVQP4KQ, seg_pos={0 0}, upload=316220]: Done uploading piece. err: <nil> Oct 8 12:55:55.453100767 (359h24m40.91519324s) [noise=true , address=145.79.227.13:4801 , piece_id=365WTSG5MCTHEZNJINWNGNULFFZJ4CBKHU32P5CAOPGU5LLW2QIA, seg_pos={0 0}, upload=316220]: Done uploading piece. err: <nil> Oct 8 12:55:55.454696335 (359h24m40.916788808s) [noise=true , address=145.79.227.5:4221 , piece_id=JHMBHZY6BHYKFAIIVJ3HIY54PBBHAQH3DCH5WV42JC4PKFIO2VXA, seg_pos={0 0}, upload=316220]: Done uploading piece. err: <nil> Oct 8 12:55:55.455046171 (359h24m40.917138644s) [noise=true , address=145.79.227.17:4972 , piece_id=4G3SFKWEPJ7JC6IJTPRO5232LPSTKYQIQMXQB2GILFPZLWAMDOJQ, seg_pos={0 0}, upload=316220]: Done uploading piece. err: <nil> Oct 8 12:55:55.458127990 (359h24m40.920220483s) [noise=true , address=145.79.227.7:4295 , piece_id=3ESCMJMPBAMJ73DSM446PEKEQZNXP53L2XZDRNFUBMECSHAFYVDA, seg_pos={0 0}, upload=316220]: Done uploading piece. err: <nil> Oct 8 12:55:55.458611371 (359h24m40.920703864s) [noise=true , address=145.79.227.3:4150 , piece_id=WOFGMBFBG5NGBOA6MFNMLR4UL3SAD5UVOEETXV4LS5NLQC2JEZDA, seg_pos={0 0}, upload=316220]: Done uploading piece. err: <nil> Oct 8 12:55:55.711027501 (359h24m41.173119974s) [noise=true , address=45.91.116.6:4974 , piece_id=KFWEUE5CUCYXYKES2B5DC7DXCNYZIFUYRH3EETNNIRXSWUMNG2NA, seg_pos={0 0}, upload=316220]: Done uploading piece. err: <nil> Oct 8 12:55:55.758245494 (359h24m41.220337967s) [noise=true , address=45.91.116.10:4083 , piece_id=SYVWQOBNBJWU34ZFPBJE3NKKGFXEXYMSVKWNQ3X22I3IFRFE53VQ, seg_pos={0 0}, upload=316220]: Done uploading piece. err: <nil> Oct 8 12:55:55.758516020 (359h24m41.220608503s) [noise=true , address=45.91.116.9:4262 , piece_id=QCVD5JFC7SBAFMQMBZVUP2NFXUSZEAOV5D6FTJ7SBSSKOTS2Q7FQ, seg_pos={0 0}, upload=316220]: Done uploading piece. err: <nil> Oct 8 12:55:55.758527291 (359h24m41.220619774s) [seg_pos={0 0}, upload=316220]: Segment reached optimal threshold of 13 pieces. Oct 8 12:55:55.758662158 (359h24m41.220754631s) [noise=true , address=77.90.31.13:4626 , piece_id=LTGV7G3MJZUFIEYZV3SS3XR2O2G2CUBIDQJ7UKRVKWV6NA7EPODQ, seg_pos={0 0}, upload=316220]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:55706->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:55706->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55706->77.90.31.13:4626: use of closed network connection Oct 8 12:55:55.758670082 (359h24m41.220762566s) [noise=true , address=77.90.31.13:4626 , piece_id=LTGV7G3MJZUFIEYZV3SS3XR2O2G2CUBIDQJ7UKRVKWV6NA7EPODQ, seg_pos={0 0}, upload=316220]: Long tail cancellation due to optimal threshold Oct 8 12:55:55.758701382 (359h24m41.220793855s) [noise=true , address=77.90.31.10:4520 , piece_id=TX7H5DQWBX6F5FENMQHDWJDDOSLR4K4XVYZGDVEROECM6ZRJR6AQ, seg_pos={0 0}, upload=316220]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:52116->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:52116->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52116->77.90.31.10:4520: use of closed network connection Oct 8 12:55:55.758710038 (359h24m41.220802521s) [noise=true , address=77.90.31.10:4520 , piece_id=TX7H5DQWBX6F5FENMQHDWJDDOSLR4K4XVYZGDVEROECM6ZRJR6AQ, seg_pos={0 0}, upload=316220]: Long tail cancellation due to optimal threshold Oct 8 12:55:55.758738262 (359h24m41.220830745s) [noise=true , address=77.90.31.16:4735 , piece_id=PSERZ5IIOBXK52H6SM6H4GMFXO24KDNNWU5GH5YWUHULGG27H5OQ, seg_pos={0 0}, upload=316220]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:33408->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:33408->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33408->77.90.31.16:4735: use of closed network connection Oct 8 12:55:55.758743452 (359h24m41.220835925s) [noise=true , address=77.90.31.16:4735 , piece_id=PSERZ5IIOBXK52H6SM6H4GMFXO24KDNNWU5GH5YWUHULGG27H5OQ, seg_pos={0 0}, upload=316220]: Long tail cancellation due to optimal threshold Oct 8 12:55:55.758757820 (359h24m41.220850293s) [seg_pos={0 0}, upload=316220]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:55:55.758825028 (359h24m41.220917501s) [upload=316220]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:55:55.758830128 (359h24m41.220922611s) [upload=316220]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:55:56.086203593 (359h24m41.548296076s) [upload=316220]: Done uploading segments... Oct 8 12:55:56.086224022 (359h24m41.548316495s) [upload=316220]: Upload finished. err: <nil> Duration:* 30.002636003s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.535032916s FAIL (2025-10-08 12:56:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.535057002s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.486811304s FAIL (2025-10-08 12:55:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.486841081s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............X........X..X..X....XX..XXX.XX..X.XX..X........] duration: 6.091068648s FAIL (2025-10-08 12:55:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ) context deadline exceeded ======LOGS====== Oct 8 12:48:14.415220350 (359h16m59.877312833s) [upload=316110]: Starting upload Oct 8 12:48:14.415286917 (359h16m59.8773794s) [upload=316110]: Done starting upload Oct 8 12:48:14.415480585 (359h16m59.877573058s) [upload=316110]: Uploading segments... Oct 8 12:48:14.415511885 (359h16m59.877604368s) [upload=316110]: Got next segment. Inline: false Oct 8 12:48:14.415545639 (359h16m59.877638122s) [upload=316110]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:48:14.415550218 (359h16m59.877642691s) [upload=316110]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:48:14.665189753 (359h17m0.127282236s) [seg_pos={0 0}, upload=316110]: Begin upload segment... Oct 8 12:48:14.665253805 (359h17m0.127346278s) [seg_pos={0 0}, upload=316110]: Done begin upload segment. Oct 8 12:48:14.665294222 (359h17m0.127386695s) [noise=true , address=45.91.116.7:4011 , piece_id=PSLYUKC4S272FLCYK5B2YVRDJKQMIIABXKG456K372JNYGZRBCRA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.665337384 (359h17m0.127429867s) [noise=true , address=45.91.116.4:4937 , piece_id=HGZMVUOX2DQF5IV6FOJYWENEKSOTVV3WPILBQZ445NDLYMPDBZQA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.665928820 (359h17m0.128021303s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666270942 (359h17m0.128363415s) [noise=true , address=45.91.116.6:4975 , piece_id=VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666599216 (359h17m0.128691689s) [noise=true , address=145.79.227.14:4840 , piece_id=MMQYGM37DKLUN7G7D2V3C6MV4JU7JWBAUQCUG2MBHCA2INUTCPIQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666877146 (359h17m0.128969619s) [noise=true , address=45.91.116.5:4298 , piece_id=JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.667141109 (359h17m0.129233602s) [noise=true , address=145.79.227.2:4114 , piece_id=TTJIY3CGDQ5SEO3T2SWJLVSWMFIF2W2X2Z5BOUCRHIQGQVGL4PBA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.667449666 (359h17m0.129542139s) [noise=true , address=145.79.227.18:4005 , piece_id=CQZBNNSPXFNAREYACXQCXMQUHLS7IQDHVXKV5XJ6BTJ3PB4KWGPA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.669911534 (359h17m0.132004027s) [noise=true , address=145.79.227.12:4767 , piece_id=VWVGJN6RBZVVUWVIRVVKI5LSGJA7HDFMK76MXHIJK4RPI6AKYRQA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.670472563 (359h17m0.132565046s) [noise=true , address=145.79.227.2:4115 , piece_id=DR56ZGOWX5S4R3NXCPYMFYP7TLAYBXMXXBTQHUFCQWU2VO2S3XAA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.670894095 (359h17m0.132986578s) [noise=true , address=145.79.227.6:4258 , piece_id=4CQHYTTGMIOD64W4N2GZALAH4S4P7WPNHYMU25MTP64SLFP6THIA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.671263718 (359h17m0.133356201s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.672089391 (359h17m0.134181874s) [noise=true , address=145.79.227.5:4221 , piece_id=43YYQOT4GIWC3656NTMDPXUDD6R7FEBYFN5UNHGLWV577U2SH2GQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.673654281 (359h17m0.135746764s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.675995769 (359h17m0.138088252s) [noise=true , address=145.79.227.17:4971 , piece_id=QB2LT2XLNEKVB7B3X7BT62L7AX7NL2F4LH3Y3NSMTVJEWRMOH4DQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.678818414 (359h17m0.140910887s) [noise=true , address=45.91.116.9:4263 , piece_id=J5UXDHXYONA67RZNHNUGFA7ZX6ANAZ2CW67PXGSDRMKKZ7SMWXIQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:16.370493184 (359h17m1.832585667s) [upload=316110]: Next returned nil segment Oct 8 12:48:16.370552197 (359h17m1.83264468s) [upload=316110]: Waiting for error group managing segments... Oct 8 12:48:16.645639979 (359h17m2.107732462s) [noise=true , address=145.79.227.14:4840 , piece_id=MMQYGM37DKLUN7G7D2V3C6MV4JU7JWBAUQCUG2MBHCA2INUTCPIQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.662240289 (359h17m2.124332762s) [noise=true , address=145.79.227.18:4005 , piece_id=CQZBNNSPXFNAREYACXQCXMQUHLS7IQDHVXKV5XJ6BTJ3PB4KWGPA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.665201317 (359h17m2.12729381s) [noise=true , address=145.79.227.2:4115 , piece_id=DR56ZGOWX5S4R3NXCPYMFYP7TLAYBXMXXBTQHUFCQWU2VO2S3XAA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.666842453 (359h17m2.128934926s) [noise=true , address=145.79.227.5:4221 , piece_id=43YYQOT4GIWC3656NTMDPXUDD6R7FEBYFN5UNHGLWV577U2SH2GQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.669086125 (359h17m2.131178608s) [noise=true , address=145.79.227.12:4767 , piece_id=VWVGJN6RBZVVUWVIRVVKI5LSGJA7HDFMK76MXHIJK4RPI6AKYRQA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.669677843 (359h17m2.131770326s) [noise=true , address=145.79.227.2:4114 , piece_id=TTJIY3CGDQ5SEO3T2SWJLVSWMFIF2W2X2Z5BOUCRHIQGQVGL4PBA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.674348055 (359h17m2.136440539s) [noise=true , address=145.79.227.6:4258 , piece_id=4CQHYTTGMIOD64W4N2GZALAH4S4P7WPNHYMU25MTP64SLFP6THIA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.675165752 (359h17m2.137258245s) [noise=true , address=145.79.227.17:4971 , piece_id=QB2LT2XLNEKVB7B3X7BT62L7AX7NL2F4LH3Y3NSMTVJEWRMOH4DQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.868106119 (359h17m2.330198602s) [noise=true , address=45.91.116.4:4937 , piece_id=HGZMVUOX2DQF5IV6FOJYWENEKSOTVV3WPILBQZ445NDLYMPDBZQA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.939659241 (359h17m2.401751724s) [noise=true , address=45.91.116.9:4263 , piece_id=J5UXDHXYONA67RZNHNUGFA7ZX6ANAZ2CW67PXGSDRMKKZ7SMWXIQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.950945673 (359h17m2.413038156s) [noise=true , address=45.91.116.6:4975 , piece_id=VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.965924876 (359h17m2.428017349s) [noise=true , address=45.91.116.5:4298 , piece_id=JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.997329036 (359h17m2.459421509s) [noise=true , address=45.91.116.7:4011 , piece_id=PSLYUKC4S272FLCYK5B2YVRDJKQMIIABXKG456K372JNYGZRBCRA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.997362129 (359h17m2.459454612s) [seg_pos={0 0}, upload=316110]: Segment reached optimal threshold of 13 pieces. Oct 8 12:48:16.997538374 (359h17m2.459630857s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997551570 (359h17m2.459644053s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997609440 (359h17m2.459701923s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997628917 (359h17m2.4597214s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997675966 (359h17m2.45976846s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997685926 (359h17m2.459778409s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997710432 (359h17m2.459802905s) [seg_pos={0 0}, upload=316110]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:48:16.997858034 (359h17m2.459950527s) [upload=316110]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:48:16.997867181 (359h17m2.459959664s) [upload=316110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:48:17.305538248 (359h17m2.767630732s) [upload=316110]: Done uploading segments... Oct 8 12:48:17.305562254 (359h17m2.767654728s) [upload=316110]: Upload finished. err: <nil> Duration: 6.091100098s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..............X........X..X..X....XX..XXX.XX..X.XX..X.......] duration: 12.712078209s FAIL (2025-10-08 12:54:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ) context deadline exceeded ======LOGS====== Oct 8 12:48:14.415220350 (359h16m59.877312833s) [upload=316110]: Starting upload Oct 8 12:48:14.415286917 (359h16m59.8773794s) [upload=316110]: Done starting upload Oct 8 12:48:14.415480585 (359h16m59.877573058s) [upload=316110]: Uploading segments... Oct 8 12:48:14.415511885 (359h16m59.877604368s) [upload=316110]: Got next segment. Inline: false Oct 8 12:48:14.415545639 (359h16m59.877638122s) [upload=316110]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:48:14.415550218 (359h16m59.877642691s) [upload=316110]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:48:14.665189753 (359h17m0.127282236s) [seg_pos={0 0}, upload=316110]: Begin upload segment... Oct 8 12:48:14.665253805 (359h17m0.127346278s) [seg_pos={0 0}, upload=316110]: Done begin upload segment. Oct 8 12:48:14.665294222 (359h17m0.127386695s) [noise=true , address=45.91.116.7:4011 , piece_id=PSLYUKC4S272FLCYK5B2YVRDJKQMIIABXKG456K372JNYGZRBCRA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.665337384 (359h17m0.127429867s) [noise=true , address=45.91.116.4:4937 , piece_id=HGZMVUOX2DQF5IV6FOJYWENEKSOTVV3WPILBQZ445NDLYMPDBZQA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.665928820 (359h17m0.128021303s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666270942 (359h17m0.128363415s) [noise=true , address=45.91.116.6:4975 , piece_id=VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666599216 (359h17m0.128691689s) [noise=true , address=145.79.227.14:4840 , piece_id=MMQYGM37DKLUN7G7D2V3C6MV4JU7JWBAUQCUG2MBHCA2INUTCPIQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666877146 (359h17m0.128969619s) [noise=true , address=45.91.116.5:4298 , piece_id=JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.667141109 (359h17m0.129233602s) [noise=true , address=145.79.227.2:4114 , piece_id=TTJIY3CGDQ5SEO3T2SWJLVSWMFIF2W2X2Z5BOUCRHIQGQVGL4PBA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.667449666 (359h17m0.129542139s) [noise=true , address=145.79.227.18:4005 , piece_id=CQZBNNSPXFNAREYACXQCXMQUHLS7IQDHVXKV5XJ6BTJ3PB4KWGPA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.669911534 (359h17m0.132004027s) [noise=true , address=145.79.227.12:4767 , piece_id=VWVGJN6RBZVVUWVIRVVKI5LSGJA7HDFMK76MXHIJK4RPI6AKYRQA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.670472563 (359h17m0.132565046s) [noise=true , address=145.79.227.2:4115 , piece_id=DR56ZGOWX5S4R3NXCPYMFYP7TLAYBXMXXBTQHUFCQWU2VO2S3XAA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.670894095 (359h17m0.132986578s) [noise=true , address=145.79.227.6:4258 , piece_id=4CQHYTTGMIOD64W4N2GZALAH4S4P7WPNHYMU25MTP64SLFP6THIA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.671263718 (359h17m0.133356201s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.672089391 (359h17m0.134181874s) [noise=true , address=145.79.227.5:4221 , piece_id=43YYQOT4GIWC3656NTMDPXUDD6R7FEBYFN5UNHGLWV577U2SH2GQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.673654281 (359h17m0.135746764s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.675995769 (359h17m0.138088252s) [noise=true , address=145.79.227.17:4971 , piece_id=QB2LT2XLNEKVB7B3X7BT62L7AX7NL2F4LH3Y3NSMTVJEWRMOH4DQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.678818414 (359h17m0.140910887s) [noise=true , address=45.91.116.9:4263 , piece_id=J5UXDHXYONA67RZNHNUGFA7ZX6ANAZ2CW67PXGSDRMKKZ7SMWXIQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:16.370493184 (359h17m1.832585667s) [upload=316110]: Next returned nil segment Oct 8 12:48:16.370552197 (359h17m1.83264468s) [upload=316110]: Waiting for error group managing segments... Oct 8 12:48:16.645639979 (359h17m2.107732462s) [noise=true , address=145.79.227.14:4840 , piece_id=MMQYGM37DKLUN7G7D2V3C6MV4JU7JWBAUQCUG2MBHCA2INUTCPIQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.662240289 (359h17m2.124332762s) [noise=true , address=145.79.227.18:4005 , piece_id=CQZBNNSPXFNAREYACXQCXMQUHLS7IQDHVXKV5XJ6BTJ3PB4KWGPA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.665201317 (359h17m2.12729381s) [noise=true , address=145.79.227.2:4115 , piece_id=DR56ZGOWX5S4R3NXCPYMFYP7TLAYBXMXXBTQHUFCQWU2VO2S3XAA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.666842453 (359h17m2.128934926s) [noise=true , address=145.79.227.5:4221 , piece_id=43YYQOT4GIWC3656NTMDPXUDD6R7FEBYFN5UNHGLWV577U2SH2GQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.669086125 (359h17m2.131178608s) [noise=true , address=145.79.227.12:4767 , piece_id=VWVGJN6RBZVVUWVIRVVKI5LSGJA7HDFMK76MXHIJK4RPI6AKYRQA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.669677843 (359h17m2.131770326s) [noise=true , address=145.79.227.2:4114 , piece_id=TTJIY3CGDQ5SEO3T2SWJLVSWMFIF2W2X2Z5BOUCRHIQGQVGL4PBA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.674348055 (359h17m2.136440539s) [noise=true , address=145.79.227.6:4258 , piece_id=4CQHYTTGMIOD64W4N2GZALAH4S4P7WPNHYMU25MTP64SLFP6THIA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.675165752 (359h17m2.137258245s) [noise=true , address=145.79.227.17:4971 , piece_id=QB2LT2XLNEKVB7B3X7BT62L7AX7NL2F4LH3Y3NSMTVJEWRMOH4DQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.868106119 (359h17m2.330198602s) [noise=true , address=45.91.116.4:4937 , piece_id=HGZMVUOX2DQF5IV6FOJYWENEKSOTVV3WPILBQZ445NDLYMPDBZQA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.939659241 (359h17m2.401751724s) [noise=true , address=45.91.116.9:4263 , piece_id=J5UXDHXYONA67RZNHNUGFA7ZX6ANAZ2CW67PXGSDRMKKZ7SMWXIQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.950945673 (359h17m2.413038156s) [noise=true , address=45.91.116.6:4975 , piece_id=VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.965924876 (359h17m2.428017349s) [noise=true , address=45.91.116.5:4298 , piece_id=JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.997329036 (359h17m2.459421509s) [noise=true , address=45.91.116.7:4011 , piece_id=PSLYUKC4S272FLCYK5B2YVRDJKQMIIABXKG456K372JNYGZRBCRA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.997362129 (359h17m2.459454612s) [seg_pos={0 0}, upload=316110]: Segment reached optimal threshold of 13 pieces. Oct 8 12:48:16.997538374 (359h17m2.459630857s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997551570 (359h17m2.459644053s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997609440 (359h17m2.459701923s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997628917 (359h17m2.4597214s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997675966 (359h17m2.45976846s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997685926 (359h17m2.459778409s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997710432 (359h17m2.459802905s) [seg_pos={0 0}, upload=316110]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:48:16.997858034 (359h17m2.459950527s) [upload=316110]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:48:16.997867181 (359h17m2.459959664s) [upload=316110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:48:17.305538248 (359h17m2.767630732s) [upload=316110]: Done uploading segments... Oct 8 12:48:17.305562254 (359h17m2.767654728s) [upload=316110]: Upload finished. err: <nil> Duration: 12.71210529s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.316919914s FAIL (2025-10-08 12:54:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.316990879s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...............X........X..X..X....XX..XXX.XX..X.XX..X......] duration: 9.716425168s FAIL (2025-10-08 12:53:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ) context deadline exceeded ======LOGS====== Oct 8 12:48:14.415220350 (359h16m59.877312833s) [upload=316110]: Starting upload Oct 8 12:48:14.415286917 (359h16m59.8773794s) [upload=316110]: Done starting upload Oct 8 12:48:14.415480585 (359h16m59.877573058s) [upload=316110]: Uploading segments... Oct 8 12:48:14.415511885 (359h16m59.877604368s) [upload=316110]: Got next segment. Inline: false Oct 8 12:48:14.415545639 (359h16m59.877638122s) [upload=316110]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:48:14.415550218 (359h16m59.877642691s) [upload=316110]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:48:14.665189753 (359h17m0.127282236s) [seg_pos={0 0}, upload=316110]: Begin upload segment... Oct 8 12:48:14.665253805 (359h17m0.127346278s) [seg_pos={0 0}, upload=316110]: Done begin upload segment. Oct 8 12:48:14.665294222 (359h17m0.127386695s) [noise=true , address=45.91.116.7:4011 , piece_id=PSLYUKC4S272FLCYK5B2YVRDJKQMIIABXKG456K372JNYGZRBCRA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.665337384 (359h17m0.127429867s) [noise=true , address=45.91.116.4:4937 , piece_id=HGZMVUOX2DQF5IV6FOJYWENEKSOTVV3WPILBQZ445NDLYMPDBZQA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.665928820 (359h17m0.128021303s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666270942 (359h17m0.128363415s) [noise=true , address=45.91.116.6:4975 , piece_id=VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666599216 (359h17m0.128691689s) [noise=true , address=145.79.227.14:4840 , piece_id=MMQYGM37DKLUN7G7D2V3C6MV4JU7JWBAUQCUG2MBHCA2INUTCPIQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666877146 (359h17m0.128969619s) [noise=true , address=45.91.116.5:4298 , piece_id=JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.667141109 (359h17m0.129233602s) [noise=true , address=145.79.227.2:4114 , piece_id=TTJIY3CGDQ5SEO3T2SWJLVSWMFIF2W2X2Z5BOUCRHIQGQVGL4PBA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.667449666 (359h17m0.129542139s) [noise=true , address=145.79.227.18:4005 , piece_id=CQZBNNSPXFNAREYACXQCXMQUHLS7IQDHVXKV5XJ6BTJ3PB4KWGPA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.669911534 (359h17m0.132004027s) [noise=true , address=145.79.227.12:4767 , piece_id=VWVGJN6RBZVVUWVIRVVKI5LSGJA7HDFMK76MXHIJK4RPI6AKYRQA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.670472563 (359h17m0.132565046s) [noise=true , address=145.79.227.2:4115 , piece_id=DR56ZGOWX5S4R3NXCPYMFYP7TLAYBXMXXBTQHUFCQWU2VO2S3XAA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.670894095 (359h17m0.132986578s) [noise=true , address=145.79.227.6:4258 , piece_id=4CQHYTTGMIOD64W4N2GZALAH4S4P7WPNHYMU25MTP64SLFP6THIA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.671263718 (359h17m0.133356201s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.672089391 (359h17m0.134181874s) [noise=true , address=145.79.227.5:4221 , piece_id=43YYQOT4GIWC3656NTMDPXUDD6R7FEBYFN5UNHGLWV577U2SH2GQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.673654281 (359h17m0.135746764s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.675995769 (359h17m0.138088252s) [noise=true , address=145.79.227.17:4971 , piece_id=QB2LT2XLNEKVB7B3X7BT62L7AX7NL2F4LH3Y3NSMTVJEWRMOH4DQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.678818414 (359h17m0.140910887s) [noise=true , address=45.91.116.9:4263 , piece_id=J5UXDHXYONA67RZNHNUGFA7ZX6ANAZ2CW67PXGSDRMKKZ7SMWXIQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:16.370493184 (359h17m1.832585667s) [upload=316110]: Next returned nil segment Oct 8 12:48:16.370552197 (359h17m1.83264468s) [upload=316110]: Waiting for error group managing segments... Oct 8 12:48:16.645639979 (359h17m2.107732462s) [noise=true , address=145.79.227.14:4840 , piece_id=MMQYGM37DKLUN7G7D2V3C6MV4JU7JWBAUQCUG2MBHCA2INUTCPIQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.662240289 (359h17m2.124332762s) [noise=true , address=145.79.227.18:4005 , piece_id=CQZBNNSPXFNAREYACXQCXMQUHLS7IQDHVXKV5XJ6BTJ3PB4KWGPA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.665201317 (359h17m2.12729381s) [noise=true , address=145.79.227.2:4115 , piece_id=DR56ZGOWX5S4R3NXCPYMFYP7TLAYBXMXXBTQHUFCQWU2VO2S3XAA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.666842453 (359h17m2.128934926s) [noise=true , address=145.79.227.5:4221 , piece_id=43YYQOT4GIWC3656NTMDPXUDD6R7FEBYFN5UNHGLWV577U2SH2GQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.669086125 (359h17m2.131178608s) [noise=true , address=145.79.227.12:4767 , piece_id=VWVGJN6RBZVVUWVIRVVKI5LSGJA7HDFMK76MXHIJK4RPI6AKYRQA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.669677843 (359h17m2.131770326s) [noise=true , address=145.79.227.2:4114 , piece_id=TTJIY3CGDQ5SEO3T2SWJLVSWMFIF2W2X2Z5BOUCRHIQGQVGL4PBA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.674348055 (359h17m2.136440539s) [noise=true , address=145.79.227.6:4258 , piece_id=4CQHYTTGMIOD64W4N2GZALAH4S4P7WPNHYMU25MTP64SLFP6THIA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.675165752 (359h17m2.137258245s) [noise=true , address=145.79.227.17:4971 , piece_id=QB2LT2XLNEKVB7B3X7BT62L7AX7NL2F4LH3Y3NSMTVJEWRMOH4DQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.868106119 (359h17m2.330198602s) [noise=true , address=45.91.116.4:4937 , piece_id=HGZMVUOX2DQF5IV6FOJYWENEKSOTVV3WPILBQZ445NDLYMPDBZQA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.939659241 (359h17m2.401751724s) [noise=true , address=45.91.116.9:4263 , piece_id=J5UXDHXYONA67RZNHNUGFA7ZX6ANAZ2CW67PXGSDRMKKZ7SMWXIQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.950945673 (359h17m2.413038156s) [noise=true , address=45.91.116.6:4975 , piece_id=VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.965924876 (359h17m2.428017349s) [noise=true , address=45.91.116.5:4298 , piece_id=JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.997329036 (359h17m2.459421509s) [noise=true , address=45.91.116.7:4011 , piece_id=PSLYUKC4S272FLCYK5B2YVRDJKQMIIABXKG456K372JNYGZRBCRA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.997362129 (359h17m2.459454612s) [seg_pos={0 0}, upload=316110]: Segment reached optimal threshold of 13 pieces. Oct 8 12:48:16.997538374 (359h17m2.459630857s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997551570 (359h17m2.459644053s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997609440 (359h17m2.459701923s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997628917 (359h17m2.4597214s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997675966 (359h17m2.45976846s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997685926 (359h17m2.459778409s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997710432 (359h17m2.459802905s) [seg_pos={0 0}, upload=316110]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:48:16.997858034 (359h17m2.459950527s) [upload=316110]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:48:16.997867181 (359h17m2.459959664s) [upload=316110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:48:17.305538248 (359h17m2.767630732s) [upload=316110]: Done uploading segments... Oct 8 12:48:17.305562254 (359h17m2.767654728s) [upload=316110]: Upload finished. err: <nil> Duration: 9.716458962s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.436558058s FAIL (2025-10-08 12:53:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.436586784s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [................X........X..X..X....XX..XXX.XX..X.XX..X.....] duration: 12.206151347s FAIL (2025-10-08 12:52:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ) context deadline exceeded ======LOGS====== Oct 8 12:48:14.415220350 (359h16m59.877312833s) [upload=316110]: Starting upload Oct 8 12:48:14.415286917 (359h16m59.8773794s) [upload=316110]: Done starting upload Oct 8 12:48:14.415480585 (359h16m59.877573058s) [upload=316110]: Uploading segments... Oct 8 12:48:14.415511885 (359h16m59.877604368s) [upload=316110]: Got next segment. Inline: false Oct 8 12:48:14.415545639 (359h16m59.877638122s) [upload=316110]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:48:14.415550218 (359h16m59.877642691s) [upload=316110]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:48:14.665189753 (359h17m0.127282236s) [seg_pos={0 0}, upload=316110]: Begin upload segment... Oct 8 12:48:14.665253805 (359h17m0.127346278s) [seg_pos={0 0}, upload=316110]: Done begin upload segment. Oct 8 12:48:14.665294222 (359h17m0.127386695s) [noise=true , address=45.91.116.7:4011 , piece_id=PSLYUKC4S272FLCYK5B2YVRDJKQMIIABXKG456K372JNYGZRBCRA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.665337384 (359h17m0.127429867s) [noise=true , address=45.91.116.4:4937 , piece_id=HGZMVUOX2DQF5IV6FOJYWENEKSOTVV3WPILBQZ445NDLYMPDBZQA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.665928820 (359h17m0.128021303s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666270942 (359h17m0.128363415s) [noise=true , address=45.91.116.6:4975 , piece_id=VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666599216 (359h17m0.128691689s) [noise=true , address=145.79.227.14:4840 , piece_id=MMQYGM37DKLUN7G7D2V3C6MV4JU7JWBAUQCUG2MBHCA2INUTCPIQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666877146 (359h17m0.128969619s) [noise=true , address=45.91.116.5:4298 , piece_id=JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.667141109 (359h17m0.129233602s) [noise=true , address=145.79.227.2:4114 , piece_id=TTJIY3CGDQ5SEO3T2SWJLVSWMFIF2W2X2Z5BOUCRHIQGQVGL4PBA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.667449666 (359h17m0.129542139s) [noise=true , address=145.79.227.18:4005 , piece_id=CQZBNNSPXFNAREYACXQCXMQUHLS7IQDHVXKV5XJ6BTJ3PB4KWGPA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.669911534 (359h17m0.132004027s) [noise=true , address=145.79.227.12:4767 , piece_id=VWVGJN6RBZVVUWVIRVVKI5LSGJA7HDFMK76MXHIJK4RPI6AKYRQA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.670472563 (359h17m0.132565046s) [noise=true , address=145.79.227.2:4115 , piece_id=DR56ZGOWX5S4R3NXCPYMFYP7TLAYBXMXXBTQHUFCQWU2VO2S3XAA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.670894095 (359h17m0.132986578s) [noise=true , address=145.79.227.6:4258 , piece_id=4CQHYTTGMIOD64W4N2GZALAH4S4P7WPNHYMU25MTP64SLFP6THIA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.671263718 (359h17m0.133356201s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.672089391 (359h17m0.134181874s) [noise=true , address=145.79.227.5:4221 , piece_id=43YYQOT4GIWC3656NTMDPXUDD6R7FEBYFN5UNHGLWV577U2SH2GQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.673654281 (359h17m0.135746764s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.675995769 (359h17m0.138088252s) [noise=true , address=145.79.227.17:4971 , piece_id=QB2LT2XLNEKVB7B3X7BT62L7AX7NL2F4LH3Y3NSMTVJEWRMOH4DQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.678818414 (359h17m0.140910887s) [noise=true , address=45.91.116.9:4263 , piece_id=J5UXDHXYONA67RZNHNUGFA7ZX6ANAZ2CW67PXGSDRMKKZ7SMWXIQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:16.370493184 (359h17m1.832585667s) [upload=316110]: Next returned nil segment Oct 8 12:48:16.370552197 (359h17m1.83264468s) [upload=316110]: Waiting for error group managing segments... Oct 8 12:48:16.645639979 (359h17m2.107732462s) [noise=true , address=145.79.227.14:4840 , piece_id=MMQYGM37DKLUN7G7D2V3C6MV4JU7JWBAUQCUG2MBHCA2INUTCPIQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.662240289 (359h17m2.124332762s) [noise=true , address=145.79.227.18:4005 , piece_id=CQZBNNSPXFNAREYACXQCXMQUHLS7IQDHVXKV5XJ6BTJ3PB4KWGPA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.665201317 (359h17m2.12729381s) [noise=true , address=145.79.227.2:4115 , piece_id=DR56ZGOWX5S4R3NXCPYMFYP7TLAYBXMXXBTQHUFCQWU2VO2S3XAA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.666842453 (359h17m2.128934926s) [noise=true , address=145.79.227.5:4221 , piece_id=43YYQOT4GIWC3656NTMDPXUDD6R7FEBYFN5UNHGLWV577U2SH2GQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.669086125 (359h17m2.131178608s) [noise=true , address=145.79.227.12:4767 , piece_id=VWVGJN6RBZVVUWVIRVVKI5LSGJA7HDFMK76MXHIJK4RPI6AKYRQA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.669677843 (359h17m2.131770326s) [noise=true , address=145.79.227.2:4114 , piece_id=TTJIY3CGDQ5SEO3T2SWJLVSWMFIF2W2X2Z5BOUCRHIQGQVGL4PBA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.674348055 (359h17m2.136440539s) [noise=true , address=145.79.227.6:4258 , piece_id=4CQHYTTGMIOD64W4N2GZALAH4S4P7WPNHYMU25MTP64SLFP6THIA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.675165752 (359h17m2.137258245s) [noise=true , address=145.79.227.17:4971 , piece_id=QB2LT2XLNEKVB7B3X7BT62L7AX7NL2F4LH3Y3NSMTVJEWRMOH4DQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.868106119 (359h17m2.330198602s) [noise=true , address=45.91.116.4:4937 , piece_id=HGZMVUOX2DQF5IV6FOJYWENEKSOTVV3WPILBQZ445NDLYMPDBZQA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.939659241 (359h17m2.401751724s) [noise=true , address=45.91.116.9:4263 , piece_id=J5UXDHXYONA67RZNHNUGFA7ZX6ANAZ2CW67PXGSDRMKKZ7SMWXIQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.950945673 (359h17m2.413038156s) [noise=true , address=45.91.116.6:4975 , piece_id=VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.965924876 (359h17m2.428017349s) [noise=true , address=45.91.116.5:4298 , piece_id=JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.997329036 (359h17m2.459421509s) [noise=true , address=45.91.116.7:4011 , piece_id=PSLYUKC4S272FLCYK5B2YVRDJKQMIIABXKG456K372JNYGZRBCRA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.997362129 (359h17m2.459454612s) [seg_pos={0 0}, upload=316110]: Segment reached optimal threshold of 13 pieces. Oct 8 12:48:16.997538374 (359h17m2.459630857s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997551570 (359h17m2.459644053s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997609440 (359h17m2.459701923s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997628917 (359h17m2.4597214s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997675966 (359h17m2.45976846s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997685926 (359h17m2.459778409s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997710432 (359h17m2.459802905s) [seg_pos={0 0}, upload=316110]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:48:16.997858034 (359h17m2.459950527s) [upload=316110]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:48:16.997867181 (359h17m2.459959664s) [upload=316110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:48:17.305538248 (359h17m2.767630732s) [upload=316110]: Done uploading segments... Oct 8 12:48:17.305562254 (359h17m2.767654728s) [upload=316110]: Upload finished. err: <nil> Duration: 12.20618526s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.21574554s FAIL (2025-10-08 12:52:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.215793551s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.................X........X..X..X....XX..XXX.XX..X.XX..X....] duration: 29.295378796s FAIL (2025-10-08 12:52:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ) context deadline exceeded ======LOGS====== Oct 8 12:48:14.415220350 (359h16m59.877312833s) [upload=316110]: Starting upload Oct 8 12:48:14.415286917 (359h16m59.8773794s) [upload=316110]: Done starting upload Oct 8 12:48:14.415480585 (359h16m59.877573058s) [upload=316110]: Uploading segments... Oct 8 12:48:14.415511885 (359h16m59.877604368s) [upload=316110]: Got next segment. Inline: false Oct 8 12:48:14.415545639 (359h16m59.877638122s) [upload=316110]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:48:14.415550218 (359h16m59.877642691s) [upload=316110]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:48:14.665189753 (359h17m0.127282236s) [seg_pos={0 0}, upload=316110]: Begin upload segment... Oct 8 12:48:14.665253805 (359h17m0.127346278s) [seg_pos={0 0}, upload=316110]: Done begin upload segment. Oct 8 12:48:14.665294222 (359h17m0.127386695s) [noise=true , address=45.91.116.7:4011 , piece_id=PSLYUKC4S272FLCYK5B2YVRDJKQMIIABXKG456K372JNYGZRBCRA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.665337384 (359h17m0.127429867s) [noise=true , address=45.91.116.4:4937 , piece_id=HGZMVUOX2DQF5IV6FOJYWENEKSOTVV3WPILBQZ445NDLYMPDBZQA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.665928820 (359h17m0.128021303s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666270942 (359h17m0.128363415s) [noise=true , address=45.91.116.6:4975 , piece_id=VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666599216 (359h17m0.128691689s) [noise=true , address=145.79.227.14:4840 , piece_id=MMQYGM37DKLUN7G7D2V3C6MV4JU7JWBAUQCUG2MBHCA2INUTCPIQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666877146 (359h17m0.128969619s) [noise=true , address=45.91.116.5:4298 , piece_id=JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.667141109 (359h17m0.129233602s) [noise=true , address=145.79.227.2:4114 , piece_id=TTJIY3CGDQ5SEO3T2SWJLVSWMFIF2W2X2Z5BOUCRHIQGQVGL4PBA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.667449666 (359h17m0.129542139s) [noise=true , address=145.79.227.18:4005 , piece_id=CQZBNNSPXFNAREYACXQCXMQUHLS7IQDHVXKV5XJ6BTJ3PB4KWGPA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.669911534 (359h17m0.132004027s) [noise=true , address=145.79.227.12:4767 , piece_id=VWVGJN6RBZVVUWVIRVVKI5LSGJA7HDFMK76MXHIJK4RPI6AKYRQA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.670472563 (359h17m0.132565046s) [noise=true , address=145.79.227.2:4115 , piece_id=DR56ZGOWX5S4R3NXCPYMFYP7TLAYBXMXXBTQHUFCQWU2VO2S3XAA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.670894095 (359h17m0.132986578s) [noise=true , address=145.79.227.6:4258 , piece_id=4CQHYTTGMIOD64W4N2GZALAH4S4P7WPNHYMU25MTP64SLFP6THIA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.671263718 (359h17m0.133356201s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.672089391 (359h17m0.134181874s) [noise=true , address=145.79.227.5:4221 , piece_id=43YYQOT4GIWC3656NTMDPXUDD6R7FEBYFN5UNHGLWV577U2SH2GQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.673654281 (359h17m0.135746764s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.675995769 (359h17m0.138088252s) [noise=true , address=145.79.227.17:4971 , piece_id=QB2LT2XLNEKVB7B3X7BT62L7AX7NL2F4LH3Y3NSMTVJEWRMOH4DQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.678818414 (359h17m0.140910887s) [noise=true , address=45.91.116.9:4263 , piece_id=J5UXDHXYONA67RZNHNUGFA7ZX6ANAZ2CW67PXGSDRMKKZ7SMWXIQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:16.370493184 (359h17m1.832585667s) [upload=316110]: Next returned nil segment Oct 8 12:48:16.370552197 (359h17m1.83264468s) [upload=316110]: Waiting for error group managing segments... Oct 8 12:48:16.645639979 (359h17m2.107732462s) [noise=true , address=145.79.227.14:4840 , piece_id=MMQYGM37DKLUN7G7D2V3C6MV4JU7JWBAUQCUG2MBHCA2INUTCPIQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.662240289 (359h17m2.124332762s) [noise=true , address=145.79.227.18:4005 , piece_id=CQZBNNSPXFNAREYACXQCXMQUHLS7IQDHVXKV5XJ6BTJ3PB4KWGPA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.665201317 (359h17m2.12729381s) [noise=true , address=145.79.227.2:4115 , piece_id=DR56ZGOWX5S4R3NXCPYMFYP7TLAYBXMXXBTQHUFCQWU2VO2S3XAA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.666842453 (359h17m2.128934926s) [noise=true , address=145.79.227.5:4221 , piece_id=43YYQOT4GIWC3656NTMDPXUDD6R7FEBYFN5UNHGLWV577U2SH2GQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.669086125 (359h17m2.131178608s) [noise=true , address=145.79.227.12:4767 , piece_id=VWVGJN6RBZVVUWVIRVVKI5LSGJA7HDFMK76MXHIJK4RPI6AKYRQA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.669677843 (359h17m2.131770326s) [noise=true , address=145.79.227.2:4114 , piece_id=TTJIY3CGDQ5SEO3T2SWJLVSWMFIF2W2X2Z5BOUCRHIQGQVGL4PBA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.674348055 (359h17m2.136440539s) [noise=true , address=145.79.227.6:4258 , piece_id=4CQHYTTGMIOD64W4N2GZALAH4S4P7WPNHYMU25MTP64SLFP6THIA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.675165752 (359h17m2.137258245s) [noise=true , address=145.79.227.17:4971 , piece_id=QB2LT2XLNEKVB7B3X7BT62L7AX7NL2F4LH3Y3NSMTVJEWRMOH4DQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.868106119 (359h17m2.330198602s) [noise=true , address=45.91.116.4:4937 , piece_id=HGZMVUOX2DQF5IV6FOJYWENEKSOTVV3WPILBQZ445NDLYMPDBZQA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.939659241 (359h17m2.401751724s) [noise=true , address=45.91.116.9:4263 , piece_id=J5UXDHXYONA67RZNHNUGFA7ZX6ANAZ2CW67PXGSDRMKKZ7SMWXIQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.950945673 (359h17m2.413038156s) [noise=true , address=45.91.116.6:4975 , piece_id=VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.965924876 (359h17m2.428017349s) [noise=true , address=45.91.116.5:4298 , piece_id=JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.997329036 (359h17m2.459421509s) [noise=true , address=45.91.116.7:4011 , piece_id=PSLYUKC4S272FLCYK5B2YVRDJKQMIIABXKG456K372JNYGZRBCRA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.997362129 (359h17m2.459454612s) [seg_pos={0 0}, upload=316110]: Segment reached optimal threshold of 13 pieces. Oct 8 12:48:16.997538374 (359h17m2.459630857s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997551570 (359h17m2.459644053s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997609440 (359h17m2.459701923s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997628917 (359h17m2.4597214s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997675966 (359h17m2.45976846s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997685926 (359h17m2.459778409s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997710432 (359h17m2.459802905s) [seg_pos={0 0}, upload=316110]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:48:16.997858034 (359h17m2.459950527s) [upload=316110]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:48:16.997867181 (359h17m2.459959664s) [upload=316110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:48:17.305538248 (359h17m2.767630732s) [upload=316110]: Done uploading segments... Oct 8 12:48:17.305562254 (359h17m2.767654728s) [upload=316110]: Upload finished. err: <nil> Duration: 29.295398773s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..................X........X..X..X....XX..XXX.XX..X.XX..X...] duration: 22.661553245s FAIL (2025-10-08 12:51:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ) context deadline exceeded ======LOGS====== Oct 8 12:48:14.415220350 (359h16m59.877312833s) [upload=316110]: Starting upload Oct 8 12:48:14.415286917 (359h16m59.8773794s) [upload=316110]: Done starting upload Oct 8 12:48:14.415480585 (359h16m59.877573058s) [upload=316110]: Uploading segments... Oct 8 12:48:14.415511885 (359h16m59.877604368s) [upload=316110]: Got next segment. Inline: false Oct 8 12:48:14.415545639 (359h16m59.877638122s) [upload=316110]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:48:14.415550218 (359h16m59.877642691s) [upload=316110]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:48:14.665189753 (359h17m0.127282236s) [seg_pos={0 0}, upload=316110]: Begin upload segment... Oct 8 12:48:14.665253805 (359h17m0.127346278s) [seg_pos={0 0}, upload=316110]: Done begin upload segment. Oct 8 12:48:14.665294222 (359h17m0.127386695s) [noise=true , address=45.91.116.7:4011 , piece_id=PSLYUKC4S272FLCYK5B2YVRDJKQMIIABXKG456K372JNYGZRBCRA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.665337384 (359h17m0.127429867s) [noise=true , address=45.91.116.4:4937 , piece_id=HGZMVUOX2DQF5IV6FOJYWENEKSOTVV3WPILBQZ445NDLYMPDBZQA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.665928820 (359h17m0.128021303s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666270942 (359h17m0.128363415s) [noise=true , address=45.91.116.6:4975 , piece_id=VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666599216 (359h17m0.128691689s) [noise=true , address=145.79.227.14:4840 , piece_id=MMQYGM37DKLUN7G7D2V3C6MV4JU7JWBAUQCUG2MBHCA2INUTCPIQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666877146 (359h17m0.128969619s) [noise=true , address=45.91.116.5:4298 , piece_id=JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.667141109 (359h17m0.129233602s) [noise=true , address=145.79.227.2:4114 , piece_id=TTJIY3CGDQ5SEO3T2SWJLVSWMFIF2W2X2Z5BOUCRHIQGQVGL4PBA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.667449666 (359h17m0.129542139s) [noise=true , address=145.79.227.18:4005 , piece_id=CQZBNNSPXFNAREYACXQCXMQUHLS7IQDHVXKV5XJ6BTJ3PB4KWGPA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.669911534 (359h17m0.132004027s) [noise=true , address=145.79.227.12:4767 , piece_id=VWVGJN6RBZVVUWVIRVVKI5LSGJA7HDFMK76MXHIJK4RPI6AKYRQA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.670472563 (359h17m0.132565046s) [noise=true , address=145.79.227.2:4115 , piece_id=DR56ZGOWX5S4R3NXCPYMFYP7TLAYBXMXXBTQHUFCQWU2VO2S3XAA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.670894095 (359h17m0.132986578s) [noise=true , address=145.79.227.6:4258 , piece_id=4CQHYTTGMIOD64W4N2GZALAH4S4P7WPNHYMU25MTP64SLFP6THIA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.671263718 (359h17m0.133356201s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.672089391 (359h17m0.134181874s) [noise=true , address=145.79.227.5:4221 , piece_id=43YYQOT4GIWC3656NTMDPXUDD6R7FEBYFN5UNHGLWV577U2SH2GQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.673654281 (359h17m0.135746764s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.675995769 (359h17m0.138088252s) [noise=true , address=145.79.227.17:4971 , piece_id=QB2LT2XLNEKVB7B3X7BT62L7AX7NL2F4LH3Y3NSMTVJEWRMOH4DQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.678818414 (359h17m0.140910887s) [noise=true , address=45.91.116.9:4263 , piece_id=J5UXDHXYONA67RZNHNUGFA7ZX6ANAZ2CW67PXGSDRMKKZ7SMWXIQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:16.370493184 (359h17m1.832585667s) [upload=316110]: Next returned nil segment Oct 8 12:48:16.370552197 (359h17m1.83264468s) [upload=316110]: Waiting for error group managing segments... Oct 8 12:48:16.645639979 (359h17m2.107732462s) [noise=true , address=145.79.227.14:4840 , piece_id=MMQYGM37DKLUN7G7D2V3C6MV4JU7JWBAUQCUG2MBHCA2INUTCPIQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.662240289 (359h17m2.124332762s) [noise=true , address=145.79.227.18:4005 , piece_id=CQZBNNSPXFNAREYACXQCXMQUHLS7IQDHVXKV5XJ6BTJ3PB4KWGPA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.665201317 (359h17m2.12729381s) [noise=true , address=145.79.227.2:4115 , piece_id=DR56ZGOWX5S4R3NXCPYMFYP7TLAYBXMXXBTQHUFCQWU2VO2S3XAA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.666842453 (359h17m2.128934926s) [noise=true , address=145.79.227.5:4221 , piece_id=43YYQOT4GIWC3656NTMDPXUDD6R7FEBYFN5UNHGLWV577U2SH2GQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.669086125 (359h17m2.131178608s) [noise=true , address=145.79.227.12:4767 , piece_id=VWVGJN6RBZVVUWVIRVVKI5LSGJA7HDFMK76MXHIJK4RPI6AKYRQA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.669677843 (359h17m2.131770326s) [noise=true , address=145.79.227.2:4114 , piece_id=TTJIY3CGDQ5SEO3T2SWJLVSWMFIF2W2X2Z5BOUCRHIQGQVGL4PBA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.674348055 (359h17m2.136440539s) [noise=true , address=145.79.227.6:4258 , piece_id=4CQHYTTGMIOD64W4N2GZALAH4S4P7WPNHYMU25MTP64SLFP6THIA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.675165752 (359h17m2.137258245s) [noise=true , address=145.79.227.17:4971 , piece_id=QB2LT2XLNEKVB7B3X7BT62L7AX7NL2F4LH3Y3NSMTVJEWRMOH4DQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.868106119 (359h17m2.330198602s) [noise=true , address=45.91.116.4:4937 , piece_id=HGZMVUOX2DQF5IV6FOJYWENEKSOTVV3WPILBQZ445NDLYMPDBZQA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.939659241 (359h17m2.401751724s) [noise=true , address=45.91.116.9:4263 , piece_id=J5UXDHXYONA67RZNHNUGFA7ZX6ANAZ2CW67PXGSDRMKKZ7SMWXIQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.950945673 (359h17m2.413038156s) [noise=true , address=45.91.116.6:4975 , piece_id=VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.965924876 (359h17m2.428017349s) [noise=true , address=45.91.116.5:4298 , piece_id=JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.997329036 (359h17m2.459421509s) [noise=true , address=45.91.116.7:4011 , piece_id=PSLYUKC4S272FLCYK5B2YVRDJKQMIIABXKG456K372JNYGZRBCRA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.997362129 (359h17m2.459454612s) [seg_pos={0 0}, upload=316110]: Segment reached optimal threshold of 13 pieces. Oct 8 12:48:16.997538374 (359h17m2.459630857s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997551570 (359h17m2.459644053s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997609440 (359h17m2.459701923s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997628917 (359h17m2.4597214s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997675966 (359h17m2.45976846s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997685926 (359h17m2.459778409s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997710432 (359h17m2.459802905s) [seg_pos={0 0}, upload=316110]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:48:16.997858034 (359h17m2.459950527s) [upload=316110]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:48:16.997867181 (359h17m2.459959664s) [upload=316110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:48:17.305538248 (359h17m2.767630732s) [upload=316110]: Done uploading segments... Oct 8 12:48:17.305562254 (359h17m2.767654728s) [upload=316110]: Upload finished. err: <nil> Duration: 22.661587642s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.186204034s FAIL (2025-10-08 12:51:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.186238669s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.199000132s FAIL (2025-10-08 12:50:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.199030258s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...................X........X..X..X....XX..XXX.XX..X.XX..X..] duration: 21.772869125s FAIL (2025-10-08 12:50:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ) context deadline exceeded ======LOGS====== Oct 8 12:48:14.415220350 (359h16m59.877312833s) [upload=316110]: Starting upload Oct 8 12:48:14.415286917 (359h16m59.8773794s) [upload=316110]: Done starting upload Oct 8 12:48:14.415480585 (359h16m59.877573058s) [upload=316110]: Uploading segments... Oct 8 12:48:14.415511885 (359h16m59.877604368s) [upload=316110]: Got next segment. Inline: false Oct 8 12:48:14.415545639 (359h16m59.877638122s) [upload=316110]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:48:14.415550218 (359h16m59.877642691s) [upload=316110]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:48:14.665189753 (359h17m0.127282236s) [seg_pos={0 0}, upload=316110]: Begin upload segment... Oct 8 12:48:14.665253805 (359h17m0.127346278s) [seg_pos={0 0}, upload=316110]: Done begin upload segment. Oct 8 12:48:14.665294222 (359h17m0.127386695s) [noise=true , address=45.91.116.7:4011 , piece_id=PSLYUKC4S272FLCYK5B2YVRDJKQMIIABXKG456K372JNYGZRBCRA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.665337384 (359h17m0.127429867s) [noise=true , address=45.91.116.4:4937 , piece_id=HGZMVUOX2DQF5IV6FOJYWENEKSOTVV3WPILBQZ445NDLYMPDBZQA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.665928820 (359h17m0.128021303s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666270942 (359h17m0.128363415s) [noise=true , address=45.91.116.6:4975 , piece_id=VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666599216 (359h17m0.128691689s) [noise=true , address=145.79.227.14:4840 , piece_id=MMQYGM37DKLUN7G7D2V3C6MV4JU7JWBAUQCUG2MBHCA2INUTCPIQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666877146 (359h17m0.128969619s) [noise=true , address=45.91.116.5:4298 , piece_id=JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.667141109 (359h17m0.129233602s) [noise=true , address=145.79.227.2:4114 , piece_id=TTJIY3CGDQ5SEO3T2SWJLVSWMFIF2W2X2Z5BOUCRHIQGQVGL4PBA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.667449666 (359h17m0.129542139s) [noise=true , address=145.79.227.18:4005 , piece_id=CQZBNNSPXFNAREYACXQCXMQUHLS7IQDHVXKV5XJ6BTJ3PB4KWGPA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.669911534 (359h17m0.132004027s) [noise=true , address=145.79.227.12:4767 , piece_id=VWVGJN6RBZVVUWVIRVVKI5LSGJA7HDFMK76MXHIJK4RPI6AKYRQA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.670472563 (359h17m0.132565046s) [noise=true , address=145.79.227.2:4115 , piece_id=DR56ZGOWX5S4R3NXCPYMFYP7TLAYBXMXXBTQHUFCQWU2VO2S3XAA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.670894095 (359h17m0.132986578s) [noise=true , address=145.79.227.6:4258 , piece_id=4CQHYTTGMIOD64W4N2GZALAH4S4P7WPNHYMU25MTP64SLFP6THIA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.671263718 (359h17m0.133356201s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.672089391 (359h17m0.134181874s) [noise=true , address=145.79.227.5:4221 , piece_id=43YYQOT4GIWC3656NTMDPXUDD6R7FEBYFN5UNHGLWV577U2SH2GQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.673654281 (359h17m0.135746764s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.675995769 (359h17m0.138088252s) [noise=true , address=145.79.227.17:4971 , piece_id=QB2LT2XLNEKVB7B3X7BT62L7AX7NL2F4LH3Y3NSMTVJEWRMOH4DQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.678818414 (359h17m0.140910887s) [noise=true , address=45.91.116.9:4263 , piece_id=J5UXDHXYONA67RZNHNUGFA7ZX6ANAZ2CW67PXGSDRMKKZ7SMWXIQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:16.370493184 (359h17m1.832585667s) [upload=316110]: Next returned nil segment Oct 8 12:48:16.370552197 (359h17m1.83264468s) [upload=316110]: Waiting for error group managing segments... Oct 8 12:48:16.645639979 (359h17m2.107732462s) [noise=true , address=145.79.227.14:4840 , piece_id=MMQYGM37DKLUN7G7D2V3C6MV4JU7JWBAUQCUG2MBHCA2INUTCPIQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.662240289 (359h17m2.124332762s) [noise=true , address=145.79.227.18:4005 , piece_id=CQZBNNSPXFNAREYACXQCXMQUHLS7IQDHVXKV5XJ6BTJ3PB4KWGPA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.665201317 (359h17m2.12729381s) [noise=true , address=145.79.227.2:4115 , piece_id=DR56ZGOWX5S4R3NXCPYMFYP7TLAYBXMXXBTQHUFCQWU2VO2S3XAA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.666842453 (359h17m2.128934926s) [noise=true , address=145.79.227.5:4221 , piece_id=43YYQOT4GIWC3656NTMDPXUDD6R7FEBYFN5UNHGLWV577U2SH2GQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.669086125 (359h17m2.131178608s) [noise=true , address=145.79.227.12:4767 , piece_id=VWVGJN6RBZVVUWVIRVVKI5LSGJA7HDFMK76MXHIJK4RPI6AKYRQA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.669677843 (359h17m2.131770326s) [noise=true , address=145.79.227.2:4114 , piece_id=TTJIY3CGDQ5SEO3T2SWJLVSWMFIF2W2X2Z5BOUCRHIQGQVGL4PBA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.674348055 (359h17m2.136440539s) [noise=true , address=145.79.227.6:4258 , piece_id=4CQHYTTGMIOD64W4N2GZALAH4S4P7WPNHYMU25MTP64SLFP6THIA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.675165752 (359h17m2.137258245s) [noise=true , address=145.79.227.17:4971 , piece_id=QB2LT2XLNEKVB7B3X7BT62L7AX7NL2F4LH3Y3NSMTVJEWRMOH4DQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.868106119 (359h17m2.330198602s) [noise=true , address=45.91.116.4:4937 , piece_id=HGZMVUOX2DQF5IV6FOJYWENEKSOTVV3WPILBQZ445NDLYMPDBZQA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.939659241 (359h17m2.401751724s) [noise=true , address=45.91.116.9:4263 , piece_id=J5UXDHXYONA67RZNHNUGFA7ZX6ANAZ2CW67PXGSDRMKKZ7SMWXIQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.950945673 (359h17m2.413038156s) [noise=true , address=45.91.116.6:4975 , piece_id=VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.965924876 (359h17m2.428017349s) [noise=true , address=45.91.116.5:4298 , piece_id=JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.997329036 (359h17m2.459421509s) [noise=true , address=45.91.116.7:4011 , piece_id=PSLYUKC4S272FLCYK5B2YVRDJKQMIIABXKG456K372JNYGZRBCRA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.997362129 (359h17m2.459454612s) [seg_pos={0 0}, upload=316110]: Segment reached optimal threshold of 13 pieces. Oct 8 12:48:16.997538374 (359h17m2.459630857s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997551570 (359h17m2.459644053s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997609440 (359h17m2.459701923s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997628917 (359h17m2.4597214s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997675966 (359h17m2.45976846s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997685926 (359h17m2.459778409s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997710432 (359h17m2.459802905s) [seg_pos={0 0}, upload=316110]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:48:16.997858034 (359h17m2.459950527s) [upload=316110]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:48:16.997867181 (359h17m2.459959664s) [upload=316110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:48:17.305538248 (359h17m2.767630732s) [upload=316110]: Done uploading segments... Oct 8 12:48:17.305562254 (359h17m2.767654728s) [upload=316110]: Upload finished. err: <nil> Duration: 21.772901729s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.189633598s FAIL (2025-10-08 12:49:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.189657804s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X...................X........X..X..X....XX..XXX.XX..X.XX..X.] duration: 29.831891014s FAIL (2025-10-08 12:49:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ) context deadline exceeded ======LOGS====== Oct 8 12:48:14.415220350 (359h16m59.877312833s) [upload=316110]: Starting upload Oct 8 12:48:14.415286917 (359h16m59.8773794s) [upload=316110]: Done starting upload Oct 8 12:48:14.415480585 (359h16m59.877573058s) [upload=316110]: Uploading segments... Oct 8 12:48:14.415511885 (359h16m59.877604368s) [upload=316110]: Got next segment. Inline: false Oct 8 12:48:14.415545639 (359h16m59.877638122s) [upload=316110]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:48:14.415550218 (359h16m59.877642691s) [upload=316110]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:48:14.665189753 (359h17m0.127282236s) [seg_pos={0 0}, upload=316110]: Begin upload segment... Oct 8 12:48:14.665253805 (359h17m0.127346278s) [seg_pos={0 0}, upload=316110]: Done begin upload segment. Oct 8 12:48:14.665294222 (359h17m0.127386695s) [noise=true , address=45.91.116.7:4011 , piece_id=PSLYUKC4S272FLCYK5B2YVRDJKQMIIABXKG456K372JNYGZRBCRA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.665337384 (359h17m0.127429867s) [noise=true , address=45.91.116.4:4937 , piece_id=HGZMVUOX2DQF5IV6FOJYWENEKSOTVV3WPILBQZ445NDLYMPDBZQA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.665928820 (359h17m0.128021303s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666270942 (359h17m0.128363415s) [noise=true , address=45.91.116.6:4975 , piece_id=VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666599216 (359h17m0.128691689s) [noise=true , address=145.79.227.14:4840 , piece_id=MMQYGM37DKLUN7G7D2V3C6MV4JU7JWBAUQCUG2MBHCA2INUTCPIQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666877146 (359h17m0.128969619s) [noise=true , address=45.91.116.5:4298 , piece_id=JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.667141109 (359h17m0.129233602s) [noise=true , address=145.79.227.2:4114 , piece_id=TTJIY3CGDQ5SEO3T2SWJLVSWMFIF2W2X2Z5BOUCRHIQGQVGL4PBA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.667449666 (359h17m0.129542139s) [noise=true , address=145.79.227.18:4005 , piece_id=CQZBNNSPXFNAREYACXQCXMQUHLS7IQDHVXKV5XJ6BTJ3PB4KWGPA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.669911534 (359h17m0.132004027s) [noise=true , address=145.79.227.12:4767 , piece_id=VWVGJN6RBZVVUWVIRVVKI5LSGJA7HDFMK76MXHIJK4RPI6AKYRQA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.670472563 (359h17m0.132565046s) [noise=true , address=145.79.227.2:4115 , piece_id=DR56ZGOWX5S4R3NXCPYMFYP7TLAYBXMXXBTQHUFCQWU2VO2S3XAA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.670894095 (359h17m0.132986578s) [noise=true , address=145.79.227.6:4258 , piece_id=4CQHYTTGMIOD64W4N2GZALAH4S4P7WPNHYMU25MTP64SLFP6THIA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.671263718 (359h17m0.133356201s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.672089391 (359h17m0.134181874s) [noise=true , address=145.79.227.5:4221 , piece_id=43YYQOT4GIWC3656NTMDPXUDD6R7FEBYFN5UNHGLWV577U2SH2GQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.673654281 (359h17m0.135746764s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.675995769 (359h17m0.138088252s) [noise=true , address=145.79.227.17:4971 , piece_id=QB2LT2XLNEKVB7B3X7BT62L7AX7NL2F4LH3Y3NSMTVJEWRMOH4DQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.678818414 (359h17m0.140910887s) [noise=true , address=45.91.116.9:4263 , piece_id=J5UXDHXYONA67RZNHNUGFA7ZX6ANAZ2CW67PXGSDRMKKZ7SMWXIQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:16.370493184 (359h17m1.832585667s) [upload=316110]: Next returned nil segment Oct 8 12:48:16.370552197 (359h17m1.83264468s) [upload=316110]: Waiting for error group managing segments... Oct 8 12:48:16.645639979 (359h17m2.107732462s) [noise=true , address=145.79.227.14:4840 , piece_id=MMQYGM37DKLUN7G7D2V3C6MV4JU7JWBAUQCUG2MBHCA2INUTCPIQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.662240289 (359h17m2.124332762s) [noise=true , address=145.79.227.18:4005 , piece_id=CQZBNNSPXFNAREYACXQCXMQUHLS7IQDHVXKV5XJ6BTJ3PB4KWGPA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.665201317 (359h17m2.12729381s) [noise=true , address=145.79.227.2:4115 , piece_id=DR56ZGOWX5S4R3NXCPYMFYP7TLAYBXMXXBTQHUFCQWU2VO2S3XAA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.666842453 (359h17m2.128934926s) [noise=true , address=145.79.227.5:4221 , piece_id=43YYQOT4GIWC3656NTMDPXUDD6R7FEBYFN5UNHGLWV577U2SH2GQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.669086125 (359h17m2.131178608s) [noise=true , address=145.79.227.12:4767 , piece_id=VWVGJN6RBZVVUWVIRVVKI5LSGJA7HDFMK76MXHIJK4RPI6AKYRQA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.669677843 (359h17m2.131770326s) [noise=true , address=145.79.227.2:4114 , piece_id=TTJIY3CGDQ5SEO3T2SWJLVSWMFIF2W2X2Z5BOUCRHIQGQVGL4PBA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.674348055 (359h17m2.136440539s) [noise=true , address=145.79.227.6:4258 , piece_id=4CQHYTTGMIOD64W4N2GZALAH4S4P7WPNHYMU25MTP64SLFP6THIA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.675165752 (359h17m2.137258245s) [noise=true , address=145.79.227.17:4971 , piece_id=QB2LT2XLNEKVB7B3X7BT62L7AX7NL2F4LH3Y3NSMTVJEWRMOH4DQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.868106119 (359h17m2.330198602s) [noise=true , address=45.91.116.4:4937 , piece_id=HGZMVUOX2DQF5IV6FOJYWENEKSOTVV3WPILBQZ445NDLYMPDBZQA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.939659241 (359h17m2.401751724s) [noise=true , address=45.91.116.9:4263 , piece_id=J5UXDHXYONA67RZNHNUGFA7ZX6ANAZ2CW67PXGSDRMKKZ7SMWXIQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.950945673 (359h17m2.413038156s) [noise=true , address=45.91.116.6:4975 , piece_id=VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.965924876 (359h17m2.428017349s) [noise=true , address=45.91.116.5:4298 , piece_id=JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.997329036 (359h17m2.459421509s) [noise=true , address=45.91.116.7:4011 , piece_id=PSLYUKC4S272FLCYK5B2YVRDJKQMIIABXKG456K372JNYGZRBCRA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.997362129 (359h17m2.459454612s) [seg_pos={0 0}, upload=316110]: Segment reached optimal threshold of 13 pieces. Oct 8 12:48:16.997538374 (359h17m2.459630857s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997551570 (359h17m2.459644053s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997609440 (359h17m2.459701923s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997628917 (359h17m2.4597214s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997675966 (359h17m2.45976846s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997685926 (359h17m2.459778409s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997710432 (359h17m2.459802905s) [seg_pos={0 0}, upload=316110]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:48:16.997858034 (359h17m2.459950527s) [upload=316110]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:48:16.997867181 (359h17m2.459959664s) [upload=316110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:48:17.305538248 (359h17m2.767630732s) [upload=316110]: Done uploading segments... Oct 8 12:48:17.305562254 (359h17m2.767654728s) [upload=316110]: Upload finished. err: <nil> Duration: 29.831921522s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.232523722s FAIL (2025-10-08 12:49:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.232549941s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX...................X........X..X..X....XX..XXX.XX..X.XX..X] duration: 30.002573145s FAIL (2025-10-08 12:48:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ) context deadline exceeded ======LOGS====== Oct 8 12:48:14.415220350 (359h16m59.877312833s) [upload=316110]: Starting upload Oct 8 12:48:14.415286917 (359h16m59.8773794s) [upload=316110]: Done starting upload Oct 8 12:48:14.415480585 (359h16m59.877573058s) [upload=316110]: Uploading segments... Oct 8 12:48:14.415511885 (359h16m59.877604368s) [upload=316110]: Got next segment. Inline: false Oct 8 12:48:14.415545639 (359h16m59.877638122s) [upload=316110]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:48:14.415550218 (359h16m59.877642691s) [upload=316110]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:48:14.665189753 (359h17m0.127282236s) [seg_pos={0 0}, upload=316110]: Begin upload segment... Oct 8 12:48:14.665253805 (359h17m0.127346278s) [seg_pos={0 0}, upload=316110]: Done begin upload segment. Oct 8 12:48:14.665294222 (359h17m0.127386695s) [noise=true , address=45.91.116.7:4011 , piece_id=PSLYUKC4S272FLCYK5B2YVRDJKQMIIABXKG456K372JNYGZRBCRA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.665337384 (359h17m0.127429867s) [noise=true , address=45.91.116.4:4937 , piece_id=HGZMVUOX2DQF5IV6FOJYWENEKSOTVV3WPILBQZ445NDLYMPDBZQA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.665928820 (359h17m0.128021303s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666270942 (359h17m0.128363415s) [noise=true , address=45.91.116.6:4975 , piece_id=VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666599216 (359h17m0.128691689s) [noise=true , address=145.79.227.14:4840 , piece_id=MMQYGM37DKLUN7G7D2V3C6MV4JU7JWBAUQCUG2MBHCA2INUTCPIQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.666877146 (359h17m0.128969619s) [noise=true , address=45.91.116.5:4298 , piece_id=JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.667141109 (359h17m0.129233602s) [noise=true , address=145.79.227.2:4114 , piece_id=TTJIY3CGDQ5SEO3T2SWJLVSWMFIF2W2X2Z5BOUCRHIQGQVGL4PBA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.667449666 (359h17m0.129542139s) [noise=true , address=145.79.227.18:4005 , piece_id=CQZBNNSPXFNAREYACXQCXMQUHLS7IQDHVXKV5XJ6BTJ3PB4KWGPA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.669911534 (359h17m0.132004027s) [noise=true , address=145.79.227.12:4767 , piece_id=VWVGJN6RBZVVUWVIRVVKI5LSGJA7HDFMK76MXHIJK4RPI6AKYRQA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.670472563 (359h17m0.132565046s) [noise=true , address=145.79.227.2:4115 , piece_id=DR56ZGOWX5S4R3NXCPYMFYP7TLAYBXMXXBTQHUFCQWU2VO2S3XAA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.670894095 (359h17m0.132986578s) [noise=true , address=145.79.227.6:4258 , piece_id=4CQHYTTGMIOD64W4N2GZALAH4S4P7WPNHYMU25MTP64SLFP6THIA, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.671263718 (359h17m0.133356201s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.672089391 (359h17m0.134181874s) [noise=true , address=145.79.227.5:4221 , piece_id=43YYQOT4GIWC3656NTMDPXUDD6R7FEBYFN5UNHGLWV577U2SH2GQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.673654281 (359h17m0.135746764s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.675995769 (359h17m0.138088252s) [noise=true , address=145.79.227.17:4971 , piece_id=QB2LT2XLNEKVB7B3X7BT62L7AX7NL2F4LH3Y3NSMTVJEWRMOH4DQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:14.678818414 (359h17m0.140910887s) [noise=true , address=45.91.116.9:4263 , piece_id=J5UXDHXYONA67RZNHNUGFA7ZX6ANAZ2CW67PXGSDRMKKZ7SMWXIQ, seg_pos={0 0}, upload=316110]: Uploading piece... Oct 8 12:48:16.370493184 (359h17m1.832585667s) [upload=316110]: Next returned nil segment Oct 8 12:48:16.370552197 (359h17m1.83264468s) [upload=316110]: Waiting for error group managing segments... Oct 8 12:48:16.645639979 (359h17m2.107732462s) [noise=true , address=145.79.227.14:4840 , piece_id=MMQYGM37DKLUN7G7D2V3C6MV4JU7JWBAUQCUG2MBHCA2INUTCPIQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.662240289 (359h17m2.124332762s) [noise=true , address=145.79.227.18:4005 , piece_id=CQZBNNSPXFNAREYACXQCXMQUHLS7IQDHVXKV5XJ6BTJ3PB4KWGPA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.665201317 (359h17m2.12729381s) [noise=true , address=145.79.227.2:4115 , piece_id=DR56ZGOWX5S4R3NXCPYMFYP7TLAYBXMXXBTQHUFCQWU2VO2S3XAA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.666842453 (359h17m2.128934926s) [noise=true , address=145.79.227.5:4221 , piece_id=43YYQOT4GIWC3656NTMDPXUDD6R7FEBYFN5UNHGLWV577U2SH2GQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.669086125 (359h17m2.131178608s) [noise=true , address=145.79.227.12:4767 , piece_id=VWVGJN6RBZVVUWVIRVVKI5LSGJA7HDFMK76MXHIJK4RPI6AKYRQA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.669677843 (359h17m2.131770326s) [noise=true , address=145.79.227.2:4114 , piece_id=TTJIY3CGDQ5SEO3T2SWJLVSWMFIF2W2X2Z5BOUCRHIQGQVGL4PBA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.674348055 (359h17m2.136440539s) [noise=true , address=145.79.227.6:4258 , piece_id=4CQHYTTGMIOD64W4N2GZALAH4S4P7WPNHYMU25MTP64SLFP6THIA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.675165752 (359h17m2.137258245s) [noise=true , address=145.79.227.17:4971 , piece_id=QB2LT2XLNEKVB7B3X7BT62L7AX7NL2F4LH3Y3NSMTVJEWRMOH4DQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.868106119 (359h17m2.330198602s) [noise=true , address=45.91.116.4:4937 , piece_id=HGZMVUOX2DQF5IV6FOJYWENEKSOTVV3WPILBQZ445NDLYMPDBZQA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.939659241 (359h17m2.401751724s) [noise=true , address=45.91.116.9:4263 , piece_id=J5UXDHXYONA67RZNHNUGFA7ZX6ANAZ2CW67PXGSDRMKKZ7SMWXIQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.950945673 (359h17m2.413038156s) [noise=true , address=45.91.116.6:4975 , piece_id=VHPQAHES75EMQ245KWRI2ABOSPKCMG4NVOKZFMGC7ST44RN642NQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.965924876 (359h17m2.428017349s) [noise=true , address=45.91.116.5:4298 , piece_id=JOKM5VGDGYLTPTDSIZHKNHUPYWHOMS5Y67LZIM4U5WEVTYR7ITXQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.997329036 (359h17m2.459421509s) [noise=true , address=45.91.116.7:4011 , piece_id=PSLYUKC4S272FLCYK5B2YVRDJKQMIIABXKG456K372JNYGZRBCRA, seg_pos={0 0}, upload=316110]: Done uploading piece. err: <nil> Oct 8 12:48:16.997362129 (359h17m2.459454612s) [seg_pos={0 0}, upload=316110]: Segment reached optimal threshold of 13 pieces. Oct 8 12:48:16.997538374 (359h17m2.459630857s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997551570 (359h17m2.459644053s) [noise=true , address=45.91.116.6:4973 , piece_id=UUVQ5C5IFPJLTAVHKCLAGHFO24LZ2HRWBCKN57637Z4DMHEHCIFQ, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997609440 (359h17m2.459701923s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997628917 (359h17m2.4597214s) [noise=true , address=77.90.31.11:4554 , piece_id=5EU5JWOOH3OSP4JT73O2RQCIQMP3ZWGYOTM6BJXUFNM3F7XDXT2A, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997675966 (359h17m2.45976846s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 8 12:48:16.997685926 (359h17m2.459778409s) [noise=true , address=77.90.31.21:4879 , piece_id=EYAS3TMTAFNPEJLFP2HK5SXZEB7IGJJEFUIICI6IP47XN5UTUSFQ, seg_pos={0 0}, upload=316110]: Long tail cancellation due to optimal threshold Oct 8 12:48:16.997710432 (359h17m2.459802905s) [seg_pos={0 0}, upload=316110]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:48:16.997858034 (359h17m2.459950527s) [upload=316110]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:48:16.997867181 (359h17m2.459959664s) [upload=316110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:48:17.305538248 (359h17m2.767630732s) [upload=316110]: Done uploading segments... Oct 8 12:48:17.305562254 (359h17m2.767654728s) [upload=316110]: Upload finished. err: <nil> Duration: 30.002599857s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.281952003s FAIL (2025-10-08 12:48:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.282030693s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.467801824s FAIL (2025-10-08 12:48:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.467861828s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.XX...................X........X..X..X....XX..XXX.XX..X.XX..] duration: 24.566635855s FAIL (2025-10-08 12:47:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: EKQSHHSXSEMQ34ITTWCFSACCZGAUDHSTCNVXCMUFPZAABJUVN4XQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: DOIHYGZG2L7OWEYS6VDDJEX7HLLZJQJTXIA2TOY27KYWBGORHPGA) context deadline exceeded ======LOGS====== ing segments... Oct 8 12:45:41.706514438 (359h14m27.168606921s) [upload=316073]: Got next segment. Inline: false Oct 8 12:45:41.706554865 (359h14m27.168647348s) [upload=316073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:45:41.706560766 (359h14m27.168653249s) [upload=316073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:45:41.957446772 (359h14m27.419539245s) [seg_pos={0 0}, upload=316073]: Begin upload segment... Oct 8 12:45:41.957518529 (359h14m27.419611022s) [seg_pos={0 0}, upload=316073]: Done begin upload segment. Oct 8 12:45:41.957550009 (359h14m27.419642492s) [noise=true , address=145.79.227.16:4909 , piece_id=EDLONA7OHKOCZJEBX7OLCMNL2ZVAXDYF6EM4XWSIFTDNN3WHREGA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.957586448 (359h14m27.419678921s) [noise=true , address=77.90.31.16:4735 , piece_id=CB6BYSHXM2YO3BCBCJHP3EHJQRALRRAG5TIN7MIJ7ZBMWHOT6XMA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.958163937 (359h14m27.42025641s) [noise=true , address=77.90.31.13:4627 , piece_id=X6WVYBXGWQ6DC5JXSKYDQEJABXKXVPOFBMMATKOPGQL3QGV5LDXQ, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.958522010 (359h14m27.420614493s) [noise=true , address=77.90.31.21:4879 , piece_id=OT4YNXPU7JE6LJ3BK3RXA4PWWLH4ZM4LCTHN5QEILU6ZWQ6N772Q, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.958890431 (359h14m27.420982904s) [noise=true , address=145.79.227.18:4006 , piece_id=BTC25AK7VM7HNGW4C5TMVUIPRXW5WKWXYN7MH2IZSS3TAF42VOPQ, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.959264664 (359h14m27.421357147s) [noise=true , address=145.79.227.6:4258 , piece_id=NCMDSV3GEWPKD76JEP3HW6H66AEOCPEWHDQYQQ4WRP4ZIIKSGVJA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.959636482 (359h14m27.421728965s) [noise=true , address=145.79.227.7:4295 , piece_id=HZE7WEPMGSYUHSGWZ77TMCGNZDF42YAPX4XODUW3ODKKIVGP54NA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.960075788 (359h14m27.422168261s) [noise=true , address=45.91.116.7:4009 , piece_id=DOIHYGZG2L7OWEYS6VDDJEX7HLLZJQJTXIA2TOY27KYWBGORHPGA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.960647687 (359h14m27.42274016s) [noise=true , address=45.91.116.5:4297 , piece_id=4N4JCCGIMLGXUZUGDWBBFJSCGT5SOW3YCQE4Z4KRWGAGZSY6SOQQ, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.962154647 (359h14m27.42424715s) [noise=true , address=45.91.116.8:4045 , piece_id=NBVXFEMZ3BPS6YDOHQOFT5MSNOJN2VW2QKMEYJW7CO7ZKJKKZP7A, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.962177190 (359h14m27.424269673s) [noise=true , address=77.90.31.19:4843 , piece_id=WL2PFSVIR4ASICJVYGZJNDANXCD6XZEIXE6WQWQOJIQULE6WH22Q, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.963649874 (359h14m27.425742347s) [noise=true , address=77.90.31.12:4590 , piece_id=EKQSHHSXSEMQ34ITTWCFSACCZGAUDHSTCNVXCMUFPZAABJUVN4XQ, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.965128609 (359h14m27.427221082s) [noise=true , address=77.90.31.14:4663 , piece_id=KKEVWO7NZVFLN3IAJIYFSKX3MVO6QNLMEOH6Q4CK7IUNZHXGS6HA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.966632754 (359h14m27.428725237s) [noise=true , address=45.91.116.4:4938 , piece_id=Q55W3P6QKFFHD6WNVJNLE4XHC45JQEMEZOU3TYLRKW5QAOQLYG2Q, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.967919394 (359h14m27.430011887s) [noise=true , address=45.91.116.14:4191 , piece_id=4I77HBJBVEDYRFIAPYDSHQ6KQWJFMN373ZORREXOONRXTZQFTL2A, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.975527091 (359h14m27.437619604s) [noise=true , address=145.79.227.5:4223 , piece_id=KW2DHAG3Y65V6X67OHHIU7W2T4HTG44ELIH352JUEXUZN6TO4LHA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:43.623368199 (359h14m29.085460672s) [upload=316073]: Next returned nil segment Oct 8 12:45:43.623388427 (359h14m29.0854809s) [upload=316073]: Waiting for error group managing segments... Oct 8 12:45:43.948096227 (359h14m29.4101887s) [noise=true , address=145.79.227.5:4223 , piece_id=KW2DHAG3Y65V6X67OHHIU7W2T4HTG44ELIH352JUEXUZN6TO4LHA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:43.948544451 (359h14m29.410636934s) [noise=true , address=145.79.227.16:4909 , piece_id=EDLONA7OHKOCZJEBX7OLCMNL2ZVAXDYF6EM4XWSIFTDNN3WHREGA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:43.953354169 (359h14m29.415446642s) [noise=true , address=145.79.227.7:4295 , piece_id=HZE7WEPMGSYUHSGWZ77TMCGNZDF42YAPX4XODUW3ODKKIVGP54NA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:43.982182144 (359h14m29.444274627s) [noise=true , address=145.79.227.6:4258 , piece_id=NCMDSV3GEWPKD76JEP3HW6H66AEOCPEWHDQYQQ4WRP4ZIIKSGVJA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.008987917 (359h14m29.471080401s) [noise=true , address=145.79.227.18:4006 , piece_id=BTC25AK7VM7HNGW4C5TMVUIPRXW5WKWXYN7MH2IZSS3TAF42VOPQ, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.235839139 (359h14m29.697931622s) [noise=true , address=45.91.116.14:4191 , piece_id=4I77HBJBVEDYRFIAPYDSHQ6KQWJFMN373ZORREXOONRXTZQFTL2A, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.251759775 (359h14m29.713852258s) [noise=true , address=45.91.116.5:4297 , piece_id=4N4JCCGIMLGXUZUGDWBBFJSCGT5SOW3YCQE4Z4KRWGAGZSY6SOQQ, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.333122456 (359h14m29.795214929s) [noise=true , address=45.91.116.7:4009 , piece_id=DOIHYGZG2L7OWEYS6VDDJEX7HLLZJQJTXIA2TOY27KYWBGORHPGA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.364791451 (359h14m29.826883924s) [noise=true , address=77.90.31.12:4590 , piece_id=EKQSHHSXSEMQ34ITTWCFSACCZGAUDHSTCNVXCMUFPZAABJUVN4XQ, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.425836724 (359h14m29.887929197s) [noise=true , address=45.91.116.8:4045 , piece_id=NBVXFEMZ3BPS6YDOHQOFT5MSNOJN2VW2QKMEYJW7CO7ZKJKKZP7A, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.606943408 (359h14m30.069035891s) [noise=true , address=77.90.31.21:4879 , piece_id=OT4YNXPU7JE6LJ3BK3RXA4PWWLH4ZM4LCTHN5QEILU6ZWQ6N772Q, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.634021781 (359h14m30.096114264s) [noise=true , address=77.90.31.13:4627 , piece_id=X6WVYBXGWQ6DC5JXSKYDQEJABXKXVPOFBMMATKOPGQL3QGV5LDXQ, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.653917656 (359h14m30.116010129s) [noise=true , address=45.91.116.4:4938 , piece_id=Q55W3P6QKFFHD6WNVJNLE4XHC45JQEMEZOU3TYLRKW5QAOQLYG2Q, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.653940330 (359h14m30.116032803s) [seg_pos={0 0}, upload=316073]: Segment reached optimal threshold of 13 pieces. Oct 8 12:45:44.654135341 (359h14m30.116227814s) [noise=true , address=77.90.31.14:4663 , piece_id=KKEVWO7NZVFLN3IAJIYFSKX3MVO6QNLMEOH6Q4CK7IUNZHXGS6HA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:54388->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:54388->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54388->77.90.31.14:4663: use of closed network connection Oct 8 12:45:44.654145901 (359h14m30.116238394s) [noise=true , address=77.90.31.14:4663 , piece_id=KKEVWO7NZVFLN3IAJIYFSKX3MVO6QNLMEOH6Q4CK7IUNZHXGS6HA, seg_pos={0 0}, upload=316073]: Long tail cancellation due to optimal threshold Oct 8 12:45:44.654171179 (359h14m30.116263652s) [noise=true , address=77.90.31.16:4735 , piece_id=CB6BYSHXM2YO3BCBCJHP3EHJQRALRRAG5TIN7MIJ7ZBMWHOT6XMA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:43876->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:43876->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43876->77.90.31.16:4735: use of closed network connection Oct 8 12:45:44.654188893 (359h14m30.116281376s) [noise=true , address=77.90.31.16:4735 , piece_id=CB6BYSHXM2YO3BCBCJHP3EHJQRALRRAG5TIN7MIJ7ZBMWHOT6XMA, seg_pos={0 0}, upload=316073]: Long tail cancellation due to optimal threshold Oct 8 12:45:44.654247856 (359h14m30.116340329s) [noise=true , address=77.90.31.19:4843 , piece_id=WL2PFSVIR4ASICJVYGZJNDANXCD6XZEIXE6WQWQOJIQULE6WH22Q, seg_pos={0 0}, upload=316073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 12:45:44.654273373 (359h14m30.116365846s) [noise=true , address=77.90.31.19:4843 , piece_id=WL2PFSVIR4ASICJVYGZJNDANXCD6XZEIXE6WQWQOJIQULE6WH22Q, seg_pos={0 0}, upload=316073]: Long tail cancellation due to optimal threshold Oct 8 12:45:44.654291017 (359h14m30.116383491s) [seg_pos={0 0}, upload=316073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:45:44.654376961 (359h14m30.116469434s) [upload=316073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:45:44.654382242 (359h14m30.116474715s) [upload=316073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:45:45.036512044 (359h14m30.498604527s) [upload=316073]: Done uploading segments... Oct 8 12:45:45.036533085 (359h14m30.498625558s) [upload=316073]: Upload finished. err: <nil> Duration: 24.566666222s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.375029134s FAIL (2025-10-08 12:47:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.37522167s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..XX...................X........X..X..X....XX..XXX.XX..X.XX.] duration: 6.047861215s FAIL (2025-10-08 12:47:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: EKQSHHSXSEMQ34ITTWCFSACCZGAUDHSTCNVXCMUFPZAABJUVN4XQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: DOIHYGZG2L7OWEYS6VDDJEX7HLLZJQJTXIA2TOY27KYWBGORHPGA) context deadline exceeded ======LOGS====== ing segments... Oct 8 12:45:41.706514438 (359h14m27.168606921s) [upload=316073]: Got next segment. Inline: false Oct 8 12:45:41.706554865 (359h14m27.168647348s) [upload=316073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:45:41.706560766 (359h14m27.168653249s) [upload=316073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:45:41.957446772 (359h14m27.419539245s) [seg_pos={0 0}, upload=316073]: Begin upload segment... Oct 8 12:45:41.957518529 (359h14m27.419611022s) [seg_pos={0 0}, upload=316073]: Done begin upload segment. Oct 8 12:45:41.957550009 (359h14m27.419642492s) [noise=true , address=145.79.227.16:4909 , piece_id=EDLONA7OHKOCZJEBX7OLCMNL2ZVAXDYF6EM4XWSIFTDNN3WHREGA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.957586448 (359h14m27.419678921s) [noise=true , address=77.90.31.16:4735 , piece_id=CB6BYSHXM2YO3BCBCJHP3EHJQRALRRAG5TIN7MIJ7ZBMWHOT6XMA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.958163937 (359h14m27.42025641s) [noise=true , address=77.90.31.13:4627 , piece_id=X6WVYBXGWQ6DC5JXSKYDQEJABXKXVPOFBMMATKOPGQL3QGV5LDXQ, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.958522010 (359h14m27.420614493s) [noise=true , address=77.90.31.21:4879 , piece_id=OT4YNXPU7JE6LJ3BK3RXA4PWWLH4ZM4LCTHN5QEILU6ZWQ6N772Q, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.958890431 (359h14m27.420982904s) [noise=true , address=145.79.227.18:4006 , piece_id=BTC25AK7VM7HNGW4C5TMVUIPRXW5WKWXYN7MH2IZSS3TAF42VOPQ, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.959264664 (359h14m27.421357147s) [noise=true , address=145.79.227.6:4258 , piece_id=NCMDSV3GEWPKD76JEP3HW6H66AEOCPEWHDQYQQ4WRP4ZIIKSGVJA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.959636482 (359h14m27.421728965s) [noise=true , address=145.79.227.7:4295 , piece_id=HZE7WEPMGSYUHSGWZ77TMCGNZDF42YAPX4XODUW3ODKKIVGP54NA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.960075788 (359h14m27.422168261s) [noise=true , address=45.91.116.7:4009 , piece_id=DOIHYGZG2L7OWEYS6VDDJEX7HLLZJQJTXIA2TOY27KYWBGORHPGA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.960647687 (359h14m27.42274016s) [noise=true , address=45.91.116.5:4297 , piece_id=4N4JCCGIMLGXUZUGDWBBFJSCGT5SOW3YCQE4Z4KRWGAGZSY6SOQQ, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.962154647 (359h14m27.42424715s) [noise=true , address=45.91.116.8:4045 , piece_id=NBVXFEMZ3BPS6YDOHQOFT5MSNOJN2VW2QKMEYJW7CO7ZKJKKZP7A, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.962177190 (359h14m27.424269673s) [noise=true , address=77.90.31.19:4843 , piece_id=WL2PFSVIR4ASICJVYGZJNDANXCD6XZEIXE6WQWQOJIQULE6WH22Q, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.963649874 (359h14m27.425742347s) [noise=true , address=77.90.31.12:4590 , piece_id=EKQSHHSXSEMQ34ITTWCFSACCZGAUDHSTCNVXCMUFPZAABJUVN4XQ, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.965128609 (359h14m27.427221082s) [noise=true , address=77.90.31.14:4663 , piece_id=KKEVWO7NZVFLN3IAJIYFSKX3MVO6QNLMEOH6Q4CK7IUNZHXGS6HA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.966632754 (359h14m27.428725237s) [noise=true , address=45.91.116.4:4938 , piece_id=Q55W3P6QKFFHD6WNVJNLE4XHC45JQEMEZOU3TYLRKW5QAOQLYG2Q, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.967919394 (359h14m27.430011887s) [noise=true , address=45.91.116.14:4191 , piece_id=4I77HBJBVEDYRFIAPYDSHQ6KQWJFMN373ZORREXOONRXTZQFTL2A, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.975527091 (359h14m27.437619604s) [noise=true , address=145.79.227.5:4223 , piece_id=KW2DHAG3Y65V6X67OHHIU7W2T4HTG44ELIH352JUEXUZN6TO4LHA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:43.623368199 (359h14m29.085460672s) [upload=316073]: Next returned nil segment Oct 8 12:45:43.623388427 (359h14m29.0854809s) [upload=316073]: Waiting for error group managing segments... Oct 8 12:45:43.948096227 (359h14m29.4101887s) [noise=true , address=145.79.227.5:4223 , piece_id=KW2DHAG3Y65V6X67OHHIU7W2T4HTG44ELIH352JUEXUZN6TO4LHA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:43.948544451 (359h14m29.410636934s) [noise=true , address=145.79.227.16:4909 , piece_id=EDLONA7OHKOCZJEBX7OLCMNL2ZVAXDYF6EM4XWSIFTDNN3WHREGA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:43.953354169 (359h14m29.415446642s) [noise=true , address=145.79.227.7:4295 , piece_id=HZE7WEPMGSYUHSGWZ77TMCGNZDF42YAPX4XODUW3ODKKIVGP54NA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:43.982182144 (359h14m29.444274627s) [noise=true , address=145.79.227.6:4258 , piece_id=NCMDSV3GEWPKD76JEP3HW6H66AEOCPEWHDQYQQ4WRP4ZIIKSGVJA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.008987917 (359h14m29.471080401s) [noise=true , address=145.79.227.18:4006 , piece_id=BTC25AK7VM7HNGW4C5TMVUIPRXW5WKWXYN7MH2IZSS3TAF42VOPQ, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.235839139 (359h14m29.697931622s) [noise=true , address=45.91.116.14:4191 , piece_id=4I77HBJBVEDYRFIAPYDSHQ6KQWJFMN373ZORREXOONRXTZQFTL2A, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.251759775 (359h14m29.713852258s) [noise=true , address=45.91.116.5:4297 , piece_id=4N4JCCGIMLGXUZUGDWBBFJSCGT5SOW3YCQE4Z4KRWGAGZSY6SOQQ, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.333122456 (359h14m29.795214929s) [noise=true , address=45.91.116.7:4009 , piece_id=DOIHYGZG2L7OWEYS6VDDJEX7HLLZJQJTXIA2TOY27KYWBGORHPGA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.364791451 (359h14m29.826883924s) [noise=true , address=77.90.31.12:4590 , piece_id=EKQSHHSXSEMQ34ITTWCFSACCZGAUDHSTCNVXCMUFPZAABJUVN4XQ, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.425836724 (359h14m29.887929197s) [noise=true , address=45.91.116.8:4045 , piece_id=NBVXFEMZ3BPS6YDOHQOFT5MSNOJN2VW2QKMEYJW7CO7ZKJKKZP7A, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.606943408 (359h14m30.069035891s) [noise=true , address=77.90.31.21:4879 , piece_id=OT4YNXPU7JE6LJ3BK3RXA4PWWLH4ZM4LCTHN5QEILU6ZWQ6N772Q, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.634021781 (359h14m30.096114264s) [noise=true , address=77.90.31.13:4627 , piece_id=X6WVYBXGWQ6DC5JXSKYDQEJABXKXVPOFBMMATKOPGQL3QGV5LDXQ, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.653917656 (359h14m30.116010129s) [noise=true , address=45.91.116.4:4938 , piece_id=Q55W3P6QKFFHD6WNVJNLE4XHC45JQEMEZOU3TYLRKW5QAOQLYG2Q, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.653940330 (359h14m30.116032803s) [seg_pos={0 0}, upload=316073]: Segment reached optimal threshold of 13 pieces. Oct 8 12:45:44.654135341 (359h14m30.116227814s) [noise=true , address=77.90.31.14:4663 , piece_id=KKEVWO7NZVFLN3IAJIYFSKX3MVO6QNLMEOH6Q4CK7IUNZHXGS6HA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:54388->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:54388->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54388->77.90.31.14:4663: use of closed network connection Oct 8 12:45:44.654145901 (359h14m30.116238394s) [noise=true , address=77.90.31.14:4663 , piece_id=KKEVWO7NZVFLN3IAJIYFSKX3MVO6QNLMEOH6Q4CK7IUNZHXGS6HA, seg_pos={0 0}, upload=316073]: Long tail cancellation due to optimal threshold Oct 8 12:45:44.654171179 (359h14m30.116263652s) [noise=true , address=77.90.31.16:4735 , piece_id=CB6BYSHXM2YO3BCBCJHP3EHJQRALRRAG5TIN7MIJ7ZBMWHOT6XMA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:43876->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:43876->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43876->77.90.31.16:4735: use of closed network connection Oct 8 12:45:44.654188893 (359h14m30.116281376s) [noise=true , address=77.90.31.16:4735 , piece_id=CB6BYSHXM2YO3BCBCJHP3EHJQRALRRAG5TIN7MIJ7ZBMWHOT6XMA, seg_pos={0 0}, upload=316073]: Long tail cancellation due to optimal threshold Oct 8 12:45:44.654247856 (359h14m30.116340329s) [noise=true , address=77.90.31.19:4843 , piece_id=WL2PFSVIR4ASICJVYGZJNDANXCD6XZEIXE6WQWQOJIQULE6WH22Q, seg_pos={0 0}, upload=316073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 12:45:44.654273373 (359h14m30.116365846s) [noise=true , address=77.90.31.19:4843 , piece_id=WL2PFSVIR4ASICJVYGZJNDANXCD6XZEIXE6WQWQOJIQULE6WH22Q, seg_pos={0 0}, upload=316073]: Long tail cancellation due to optimal threshold Oct 8 12:45:44.654291017 (359h14m30.116383491s) [seg_pos={0 0}, upload=316073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:45:44.654376961 (359h14m30.116469434s) [upload=316073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:45:44.654382242 (359h14m30.116474715s) [upload=316073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:45:45.036512044 (359h14m30.498604527s) [upload=316073]: Done uploading segments... Oct 8 12:45:45.036533085 (359h14m30.498625558s) [upload=316073]: Upload finished. err: <nil> Duration: 6.047890472s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.299423058s FAIL (2025-10-08 12:46:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.299447675s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.186242566s FAIL (2025-10-08 12:46:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.186271019s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...XX...................X........X..X..X....XX..XXX.XX..X.XX] duration: 30.002645125s FAIL (2025-10-08 12:46:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: EKQSHHSXSEMQ34ITTWCFSACCZGAUDHSTCNVXCMUFPZAABJUVN4XQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: DOIHYGZG2L7OWEYS6VDDJEX7HLLZJQJTXIA2TOY27KYWBGORHPGA) context deadline exceeded ======LOGS====== ing segments... Oct 8 12:45:41.706514438 (359h14m27.168606921s) [upload=316073]: Got next segment. Inline: false Oct 8 12:45:41.706554865 (359h14m27.168647348s) [upload=316073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:45:41.706560766 (359h14m27.168653249s) [upload=316073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:45:41.957446772 (359h14m27.419539245s) [seg_pos={0 0}, upload=316073]: Begin upload segment... Oct 8 12:45:41.957518529 (359h14m27.419611022s) [seg_pos={0 0}, upload=316073]: Done begin upload segment. Oct 8 12:45:41.957550009 (359h14m27.419642492s) [noise=true , address=145.79.227.16:4909 , piece_id=EDLONA7OHKOCZJEBX7OLCMNL2ZVAXDYF6EM4XWSIFTDNN3WHREGA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.957586448 (359h14m27.419678921s) [noise=true , address=77.90.31.16:4735 , piece_id=CB6BYSHXM2YO3BCBCJHP3EHJQRALRRAG5TIN7MIJ7ZBMWHOT6XMA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.958163937 (359h14m27.42025641s) [noise=true , address=77.90.31.13:4627 , piece_id=X6WVYBXGWQ6DC5JXSKYDQEJABXKXVPOFBMMATKOPGQL3QGV5LDXQ, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.958522010 (359h14m27.420614493s) [noise=true , address=77.90.31.21:4879 , piece_id=OT4YNXPU7JE6LJ3BK3RXA4PWWLH4ZM4LCTHN5QEILU6ZWQ6N772Q, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.958890431 (359h14m27.420982904s) [noise=true , address=145.79.227.18:4006 , piece_id=BTC25AK7VM7HNGW4C5TMVUIPRXW5WKWXYN7MH2IZSS3TAF42VOPQ, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.959264664 (359h14m27.421357147s) [noise=true , address=145.79.227.6:4258 , piece_id=NCMDSV3GEWPKD76JEP3HW6H66AEOCPEWHDQYQQ4WRP4ZIIKSGVJA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.959636482 (359h14m27.421728965s) [noise=true , address=145.79.227.7:4295 , piece_id=HZE7WEPMGSYUHSGWZ77TMCGNZDF42YAPX4XODUW3ODKKIVGP54NA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.960075788 (359h14m27.422168261s) [noise=true , address=45.91.116.7:4009 , piece_id=DOIHYGZG2L7OWEYS6VDDJEX7HLLZJQJTXIA2TOY27KYWBGORHPGA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.960647687 (359h14m27.42274016s) [noise=true , address=45.91.116.5:4297 , piece_id=4N4JCCGIMLGXUZUGDWBBFJSCGT5SOW3YCQE4Z4KRWGAGZSY6SOQQ, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.962154647 (359h14m27.42424715s) [noise=true , address=45.91.116.8:4045 , piece_id=NBVXFEMZ3BPS6YDOHQOFT5MSNOJN2VW2QKMEYJW7CO7ZKJKKZP7A, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.962177190 (359h14m27.424269673s) [noise=true , address=77.90.31.19:4843 , piece_id=WL2PFSVIR4ASICJVYGZJNDANXCD6XZEIXE6WQWQOJIQULE6WH22Q, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.963649874 (359h14m27.425742347s) [noise=true , address=77.90.31.12:4590 , piece_id=EKQSHHSXSEMQ34ITTWCFSACCZGAUDHSTCNVXCMUFPZAABJUVN4XQ, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.965128609 (359h14m27.427221082s) [noise=true , address=77.90.31.14:4663 , piece_id=KKEVWO7NZVFLN3IAJIYFSKX3MVO6QNLMEOH6Q4CK7IUNZHXGS6HA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.966632754 (359h14m27.428725237s) [noise=true , address=45.91.116.4:4938 , piece_id=Q55W3P6QKFFHD6WNVJNLE4XHC45JQEMEZOU3TYLRKW5QAOQLYG2Q, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.967919394 (359h14m27.430011887s) [noise=true , address=45.91.116.14:4191 , piece_id=4I77HBJBVEDYRFIAPYDSHQ6KQWJFMN373ZORREXOONRXTZQFTL2A, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:41.975527091 (359h14m27.437619604s) [noise=true , address=145.79.227.5:4223 , piece_id=KW2DHAG3Y65V6X67OHHIU7W2T4HTG44ELIH352JUEXUZN6TO4LHA, seg_pos={0 0}, upload=316073]: Uploading piece... Oct 8 12:45:43.623368199 (359h14m29.085460672s) [upload=316073]: Next returned nil segment Oct 8 12:45:43.623388427 (359h14m29.0854809s) [upload=316073]: Waiting for error group managing segments... Oct 8 12:45:43.948096227 (359h14m29.4101887s) [noise=true , address=145.79.227.5:4223 , piece_id=KW2DHAG3Y65V6X67OHHIU7W2T4HTG44ELIH352JUEXUZN6TO4LHA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:43.948544451 (359h14m29.410636934s) [noise=true , address=145.79.227.16:4909 , piece_id=EDLONA7OHKOCZJEBX7OLCMNL2ZVAXDYF6EM4XWSIFTDNN3WHREGA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:43.953354169 (359h14m29.415446642s) [noise=true , address=145.79.227.7:4295 , piece_id=HZE7WEPMGSYUHSGWZ77TMCGNZDF42YAPX4XODUW3ODKKIVGP54NA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:43.982182144 (359h14m29.444274627s) [noise=true , address=145.79.227.6:4258 , piece_id=NCMDSV3GEWPKD76JEP3HW6H66AEOCPEWHDQYQQ4WRP4ZIIKSGVJA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.008987917 (359h14m29.471080401s) [noise=true , address=145.79.227.18:4006 , piece_id=BTC25AK7VM7HNGW4C5TMVUIPRXW5WKWXYN7MH2IZSS3TAF42VOPQ, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.235839139 (359h14m29.697931622s) [noise=true , address=45.91.116.14:4191 , piece_id=4I77HBJBVEDYRFIAPYDSHQ6KQWJFMN373ZORREXOONRXTZQFTL2A, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.251759775 (359h14m29.713852258s) [noise=true , address=45.91.116.5:4297 , piece_id=4N4JCCGIMLGXUZUGDWBBFJSCGT5SOW3YCQE4Z4KRWGAGZSY6SOQQ, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.333122456 (359h14m29.795214929s) [noise=true , address=45.91.116.7:4009 , piece_id=DOIHYGZG2L7OWEYS6VDDJEX7HLLZJQJTXIA2TOY27KYWBGORHPGA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.364791451 (359h14m29.826883924s) [noise=true , address=77.90.31.12:4590 , piece_id=EKQSHHSXSEMQ34ITTWCFSACCZGAUDHSTCNVXCMUFPZAABJUVN4XQ, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.425836724 (359h14m29.887929197s) [noise=true , address=45.91.116.8:4045 , piece_id=NBVXFEMZ3BPS6YDOHQOFT5MSNOJN2VW2QKMEYJW7CO7ZKJKKZP7A, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.606943408 (359h14m30.069035891s) [noise=true , address=77.90.31.21:4879 , piece_id=OT4YNXPU7JE6LJ3BK3RXA4PWWLH4ZM4LCTHN5QEILU6ZWQ6N772Q, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.634021781 (359h14m30.096114264s) [noise=true , address=77.90.31.13:4627 , piece_id=X6WVYBXGWQ6DC5JXSKYDQEJABXKXVPOFBMMATKOPGQL3QGV5LDXQ, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.653917656 (359h14m30.116010129s) [noise=true , address=45.91.116.4:4938 , piece_id=Q55W3P6QKFFHD6WNVJNLE4XHC45JQEMEZOU3TYLRKW5QAOQLYG2Q, seg_pos={0 0}, upload=316073]: Done uploading piece. err: <nil> Oct 8 12:45:44.653940330 (359h14m30.116032803s) [seg_pos={0 0}, upload=316073]: Segment reached optimal threshold of 13 pieces. Oct 8 12:45:44.654135341 (359h14m30.116227814s) [noise=true , address=77.90.31.14:4663 , piece_id=KKEVWO7NZVFLN3IAJIYFSKX3MVO6QNLMEOH6Q4CK7IUNZHXGS6HA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:54388->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:54388->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54388->77.90.31.14:4663: use of closed network connection Oct 8 12:45:44.654145901 (359h14m30.116238394s) [noise=true , address=77.90.31.14:4663 , piece_id=KKEVWO7NZVFLN3IAJIYFSKX3MVO6QNLMEOH6Q4CK7IUNZHXGS6HA, seg_pos={0 0}, upload=316073]: Long tail cancellation due to optimal threshold Oct 8 12:45:44.654171179 (359h14m30.116263652s) [noise=true , address=77.90.31.16:4735 , piece_id=CB6BYSHXM2YO3BCBCJHP3EHJQRALRRAG5TIN7MIJ7ZBMWHOT6XMA, seg_pos={0 0}, upload=316073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:43876->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:43876->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43876->77.90.31.16:4735: use of closed network connection Oct 8 12:45:44.654188893 (359h14m30.116281376s) [noise=true , address=77.90.31.16:4735 , piece_id=CB6BYSHXM2YO3BCBCJHP3EHJQRALRRAG5TIN7MIJ7ZBMWHOT6XMA, seg_pos={0 0}, upload=316073]: Long tail cancellation due to optimal threshold Oct 8 12:45:44.654247856 (359h14m30.116340329s) [noise=true , address=77.90.31.19:4843 , piece_id=WL2PFSVIR4ASICJVYGZJNDANXCD6XZEIXE6WQWQOJIQULE6WH22Q, seg_pos={0 0}, upload=316073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 12:45:44.654273373 (359h14m30.116365846s) [noise=true , address=77.90.31.19:4843 , piece_id=WL2PFSVIR4ASICJVYGZJNDANXCD6XZEIXE6WQWQOJIQULE6WH22Q, seg_pos={0 0}, upload=316073]: Long tail cancellation due to optimal threshold Oct 8 12:45:44.654291017 (359h14m30.116383491s) [seg_pos={0 0}, upload=316073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:45:44.654376961 (359h14m30.116469434s) [upload=316073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:45:44.654382242 (359h14m30.116474715s) [upload=316073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:45:45.036512044 (359h14m30.498604527s) [upload=316073]: Done uploading segments... Oct 8 12:45:45.036533085 (359h14m30.498625558s) [upload=316073]: Upload finished. err: <nil> Duration: 30.002672207s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.27310477s FAIL (2025-10-08 12:45:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.273131511s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....XX...................X........X..X..X....XX..XXX.XX..X.X] duration: 30.001788844s FAIL (2025-10-08 12:45:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 7NLZTOBUQUVX5EUJFZYBAHVTQVSRVWX2OP6436IPWWZO2DJB4K6A) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: B6EQE6EVALXHWJ6QY36DL3ERYWXBET556ZKGR5PMWHMCEGKPJKHQ) context deadline exceeded ======LOGS====== 8 12:44:31.528243296 (359h13m16.990335779s) [upload=316055]: Starting upload Oct 8 12:44:31.528388903 (359h13m16.990481396s) [upload=316055]: Done starting upload Oct 8 12:44:31.528582742 (359h13m16.990675215s) [upload=316055]: Uploading segments... Oct 8 12:44:31.528757214 (359h13m16.990849697s) [upload=316055]: Got next segment. Inline: false Oct 8 12:44:31.528825414 (359h13m16.990917897s) [upload=316055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:44:31.528833208 (359h13m16.990925691s) [upload=316055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:44:31.787633148 (359h13m17.249725631s) [seg_pos={0 0}, upload=316055]: Begin upload segment... Oct 8 12:44:31.787703211 (359h13m17.249795694s) [seg_pos={0 0}, upload=316055]: Done begin upload segment. Oct 8 12:44:31.787740532 (359h13m17.249833005s) [noise=true , address=45.91.116.12:4227 , piece_id=77XDWJ3TXY65TBEN3KKRM3ZU3IPGFU27FOQJCUTWO7VR72OPKP4Q, seg_pos={0 0}, upload=316055]: Uploading piece... Oct 8 12:44:31.787757804 (359h13m17.249850307s) [noise=true , address=45.91.116.5:4299 , piece_id=GLO2NEE6NKTVSR3IYIM7RPG5T3SWSJYV5IMGDJZWOSIECHZALSDA, seg_pos={0 0}, upload=316055]: Uploading piece... Oct 8 12:44:31.788409145 (359h13m17.250501628s) [noise=true , address=45.91.116.14:4189 , piece_id=XR7PQJIXZF34MP3OKTZSORDPX33E7ZYSWBUEGQGPNGPKDKAJN3JQ, seg_pos={0 0}, upload=316055]: Uploading piece... Oct 8 12:44:31.788518874 (359h13m17.250611357s) [noise=true , address=145.79.227.11:4729 , piece_id=BXSOOLKN2C5FR7BCU7EFE2C5BHM3IUM6KK5COX4X3INLGE7FFFYA, seg_pos={0 0}, upload=316055]: Uploading piece... Oct 8 12:44:31.788892876 (359h13m17.250985359s) [noise=true , address=145.79.227.4:4187 , piece_id=3DVJOCPE6JFKRULBKVLNKQAZPGOOZKNB654LSTACBXY76KIFJ2TQ, seg_pos={0 0}, upload=316055]: Uploading piece... Oct 8 12:44:31.788944153 (359h13m17.251036636s) [noise=true , address=145.79.227.15:4874 , piece_id=625KKG6D5SRMC7C3O5ATODFNGW4MCABYA4EH5GEYF3KKQ7BDKGPA, seg_pos={0 0}, upload=316055]: Uploading piece... Oct 8 12:44:31.789327143 (359h13m17.251419616s) [noise=true , address=45.91.116.11:4119 , piece_id=T7CSOS6M3TKUGOXKJFCHQHTHZHWOYIFRXXJNP336RCMZYFWNWTZA, seg_pos={0 0}, upload=316055]: Uploading piece... Oct 8 12:44:31.789482509 (359h13m17.251574992s) [noise=true , address=45.91.116.7:4010 , piece_id=6GPAVQ7LDYFBNRSHSWHJAHUVOTLAZNXSAIGWKDQA7R3JWAMYOK5A, seg_pos={0 0}, upload=316055]: Uploading piece... Oct 8 12:44:31.789676037 (359h13m17.25176852s) [noise=true , address=77.90.31.18:4805 , piece_id=GWNITOEBO56LWZ4S3M5I2WGJV6IJM3V4D4SXZIXBALTO4DEFGBQA, seg_pos={0 0}, upload=316055]: Uploading piece... Oct 8 12:44:31.790055039 (359h13m17.252147522s) [noise=true , address=45.91.116.9:4262 , piece_id=LEXYZPZUTPZGSPIVHSBGYS2FAUNDUOTYBULRXUSIVMA5BHLBYFZA, seg_pos={0 0}, upload=316055]: Uploading piece... Oct 8 12:44:31.790113370 (359h13m17.252205843s) [noise=true , address=145.79.227.14:4840 , piece_id=PQNWALE6ED3JZVYWCIX3ZXT42IFXZ3MHJZAEJKGTIZLFTSFRWF2Q, seg_pos={0 0}, upload=316055]: Uploading piece... Oct 8 12:44:31.790460581 (359h13m17.252553064s) [noise=true , address=45.91.116.6:4973 , piece_id=UAAVK3JJDI5YNL4XUBPDTCJPGLLTO6LC2A5KTM2BFM4CYNLVUOLQ, seg_pos={0 0}, upload=316055]: Uploading piece... Oct 8 12:44:31.790601409 (359h13m17.252693892s) [noise=true , address=77.90.31.21:4879 , piece_id=ZYQHEW525MIKCURZ7LOEOV4XGF5UWORNUF6Q37T4GKFDQMHJCT6Q, seg_pos={0 0}, upload=316055]: Uploading piece... Oct 8 12:44:31.790823632 (359h13m17.252916105s) [noise=true , address=45.91.116.14:4191 , piece_id=B6EQE6EVALXHWJ6QY36DL3ERYWXBET556ZKGR5PMWHMCEGKPJKHQ, seg_pos={0 0}, upload=316055]: Uploading piece... Oct 8 12:44:31.791116400 (359h13m17.253208893s) [noise=true , address=145.79.227.12:4767 , piece_id=LKBAWOW2US4K5TLAUGDTA3KB2RJ5VFHIWPYZC7F55BX4WEE5TOBA, seg_pos={0 0}, upload=316055]: Uploading piece... Oct 8 12:44:31.791710411 (359h13m17.253802894s) [noise=true , address=45.91.116.6:4975 , piece_id=7NLZTOBUQUVX5EUJFZYBAHVTQVSRVWX2OP6436IPWWZO2DJB4K6A, seg_pos={0 0}, upload=316055]: Uploading piece... Oct 8 12:44:33.482023142 (359h13m18.944115625s) [upload=316055]: Next returned nil segment Oct 8 12:44:33.482046006 (359h13m18.944138489s) [upload=316055]: Waiting for error group managing segments... Oct 8 12:44:33.765562831 (359h13m19.227655314s) [noise=true , address=145.79.227.14:4840 , piece_id=PQNWALE6ED3JZVYWCIX3ZXT42IFXZ3MHJZAEJKGTIZLFTSFRWF2Q, seg_pos={0 0}, upload=316055]: Done uploading piece. err: <nil> Oct 8 12:44:33.773890237 (359h13m19.23598272s) [noise=true , address=145.79.227.4:4187 , piece_id=3DVJOCPE6JFKRULBKVLNKQAZPGOOZKNB654LSTACBXY76KIFJ2TQ, seg_pos={0 0}, upload=316055]: Done uploading piece. err: <nil> Oct 8 12:44:33.783787694 (359h13m19.245880227s) [noise=true , address=145.79.227.11:4729 , piece_id=BXSOOLKN2C5FR7BCU7EFE2C5BHM3IUM6KK5COX4X3INLGE7FFFYA, seg_pos={0 0}, upload=316055]: Done uploading piece. err: <nil> Oct 8 12:44:33.791519156 (359h13m19.253611629s) [noise=true , address=145.79.227.15:4874 , piece_id=625KKG6D5SRMC7C3O5ATODFNGW4MCABYA4EH5GEYF3KKQ7BDKGPA, seg_pos={0 0}, upload=316055]: Done uploading piece. err: <nil> Oct 8 12:44:33.793003112 (359h13m19.255095585s) [noise=true , address=145.79.227.12:4767 , piece_id=LKBAWOW2US4K5TLAUGDTA3KB2RJ5VFHIWPYZC7F55BX4WEE5TOBA, seg_pos={0 0}, upload=316055]: Done uploading piece. err: <nil> Oct 8 12:44:34.033239602 (359h13m19.495332075s) [noise=true , address=45.91.116.6:4973 , piece_id=UAAVK3JJDI5YNL4XUBPDTCJPGLLTO6LC2A5KTM2BFM4CYNLVUOLQ, seg_pos={0 0}, upload=316055]: Done uploading piece. err: <nil> Oct 8 12:44:34.036713838 (359h13m19.498806321s) [noise=true , address=45.91.116.11:4119 , piece_id=T7CSOS6M3TKUGOXKJFCHQHTHZHWOYIFRXXJNP336RCMZYFWNWTZA, seg_pos={0 0}, upload=316055]: Done uploading piece. err: <nil> Oct 8 12:44:34.038556316 (359h13m19.500648799s) [noise=true , address=45.91.116.9:4262 , piece_id=LEXYZPZUTPZGSPIVHSBGYS2FAUNDUOTYBULRXUSIVMA5BHLBYFZA, seg_pos={0 0}, upload=316055]: Done uploading piece. err: <nil> Oct 8 12:44:34.064331186 (359h13m19.526423669s) [noise=true , address=45.91.116.12:4227 , piece_id=77XDWJ3TXY65TBEN3KKRM3ZU3IPGFU27FOQJCUTWO7VR72OPKP4Q, seg_pos={0 0}, upload=316055]: Done uploading piece. err: <nil> Oct 8 12:44:34.091421483 (359h13m19.553513956s) [noise=true , address=45.91.116.14:4191 , piece_id=B6EQE6EVALXHWJ6QY36DL3ERYWXBET556ZKGR5PMWHMCEGKPJKHQ, seg_pos={0 0}, upload=316055]: Done uploading piece. err: <nil> Oct 8 12:44:34.099226705 (359h13m19.561319188s) [noise=true , address=45.91.116.6:4975 , piece_id=7NLZTOBUQUVX5EUJFZYBAHVTQVSRVWX2OP6436IPWWZO2DJB4K6A, seg_pos={0 0}, upload=316055]: Done uploading piece. err: <nil> Oct 8 12:44:34.118766022 (359h13m19.580858495s) [noise=true , address=77.90.31.21:4879 , piece_id=ZYQHEW525MIKCURZ7LOEOV4XGF5UWORNUF6Q37T4GKFDQMHJCT6Q, seg_pos={0 0}, upload=316055]: Done uploading piece. err: <nil> Oct 8 12:44:34.353000151 (359h13m19.815092624s) [noise=true , address=77.90.31.18:4805 , piece_id=GWNITOEBO56LWZ4S3M5I2WGJV6IJM3V4D4SXZIXBALTO4DEFGBQA, seg_pos={0 0}, upload=316055]: Done uploading piece. err: <nil> Oct 8 12:44:34.353028586 (359h13m19.815121089s) [seg_pos={0 0}, upload=316055]: Segment reached optimal threshold of 13 pieces. Oct 8 12:44:34.353146912 (359h13m19.815239385s) [noise=true , address=45.91.116.7:4010 , piece_id=6GPAVQ7LDYFBNRSHSWHJAHUVOTLAZNXSAIGWKDQA7R3JWAMYOK5A, seg_pos={0 0}, upload=316055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 8 12:44:34.353155598 (359h13m19.815248071s) [noise=true , address=45.91.116.7:4010 , piece_id=6GPAVQ7LDYFBNRSHSWHJAHUVOTLAZNXSAIGWKDQA7R3JWAMYOK5A, seg_pos={0 0}, upload=316055]: Long tail cancellation due to optimal threshold Oct 8 12:44:34.353267781 (359h13m19.815360254s) [noise=true , address=45.91.116.14:4189 , piece_id=XR7PQJIXZF34MP3OKTZSORDPX33E7ZYSWBUEGQGPNGPKDKAJN3JQ, seg_pos={0 0}, upload=316055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Oct 8 12:44:34.353276658 (359h13m19.815369141s) [noise=true , address=45.91.116.14:4189 , piece_id=XR7PQJIXZF34MP3OKTZSORDPX33E7ZYSWBUEGQGPNGPKDKAJN3JQ, seg_pos={0 0}, upload=316055]: Long tail cancellation due to optimal threshold Oct 8 12:44:34.353329990 (359h13m19.815422483s) [noise=true , address=45.91.116.5:4299 , piece_id=GLO2NEE6NKTVSR3IYIM7RPG5T3SWSJYV5IMGDJZWOSIECHZALSDA, seg_pos={0 0}, upload=316055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:58296->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:58296->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58296->45.91.116.5:4299: use of closed network connection Oct 8 12:44:34.353344016 (359h13m19.815436489s) [noise=true , address=45.91.116.5:4299 , piece_id=GLO2NEE6NKTVSR3IYIM7RPG5T3SWSJYV5IMGDJZWOSIECHZALSDA, seg_pos={0 0}, upload=316055]: Long tail cancellation due to optimal threshold Oct 8 12:44:34.353363143 (359h13m19.815455626s) [seg_pos={0 0}, upload=316055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:44:34.353460809 (359h13m19.815553282s) [upload=316055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:44:34.353466520 (359h13m19.815559003s) [upload=316055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:44:34.665163956 (359h13m20.127256439s) [upload=316055]: Done uploading segments... Oct 8 12:44:34.665186038 (359h13m20.127278511s) [upload=316055]: Upload finished. err: <nil> Duration: 30.001816046s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.18944693s FAIL (2025-10-08 12:44:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.189466026s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.235686359s FAIL (2025-10-08 12:44:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.235717768s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.161062963s FAIL (2025-10-08 12:44:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.161091847s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....XX...................X........X..X..X....XX..XXX.XX..X.] duration: 6.276061182s FAIL (2025-10-08 12:43:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: D3WYKJT2QUDYZG4DMDW37GRHR4TALS3R42XOHKC3RL6VCNTR6D2Q) context deadline exceeded; piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: TGIPQUVF32I7TKSHMTUM2EK5OD43JVKFOREHL5JTFLFL25LN6QOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: BD3EFMJ57GUEBFSEOAXPZNBUBFESQ7ONWDVND2ALBCHB6KEOYU7Q) context deadline exceeded ======LOGS====== t 8 12:43:03.793187630 (359h11m49.255280133s) [upload=316035]: Starting upload Oct 8 12:43:03.793285647 (359h11m49.25537813s) [upload=316035]: Done starting upload Oct 8 12:43:03.793485377 (359h11m49.25557786s) [upload=316035]: Uploading segments... Oct 8 12:43:03.793684276 (359h11m49.255776759s) [upload=316035]: Got next segment. Inline: false Oct 8 12:43:03.793763357 (359h11m49.25585584s) [upload=316035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:43:03.793770259 (359h11m49.255862733s) [upload=316035]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:43:04.052805806 (359h11m49.514898289s) [seg_pos={0 0}, upload=316035]: Begin upload segment... Oct 8 12:43:04.052884667 (359h11m49.51497715s) [seg_pos={0 0}, upload=316035]: Done begin upload segment. Oct 8 12:43:04.052929842 (359h11m49.515022365s) [noise=true , address=77.90.31.16:4734 , piece_id=TGIPQUVF32I7TKSHMTUM2EK5OD43JVKFOREHL5JTFLFL25LN6QOA, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.054231500 (359h11m49.516323984s) [noise=true , address=77.90.31.19:4843 , piece_id=IJYZPVKVLJQSJK4WGXNJP2TBXRKYNNLNUEEF6UKGMVWWFSOZ7ZQQ, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.054375856 (359h11m49.516468339s) [noise=true , address=77.90.31.21:4878 , piece_id=7LJVVGM6XY7J7XVZKZESOVPRS7ACGYDV7F3QV6TNEIOKA2X6ZEJA, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.054786848 (359h11m49.516879331s) [noise=true , address=145.79.227.5:4223 , piece_id=5J77YDYAK5W42D6B72C5EHXVUR56LZY6SRTFGF2SO4O3ID42EQPA, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.055181370 (359h11m49.517273853s) [noise=true , address=77.90.31.17:4770 , piece_id=BD3EFMJ57GUEBFSEOAXPZNBUBFESQ7ONWDVND2ALBCHB6KEOYU7Q, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.055199965 (359h11m49.517292448s) [noise=true , address=145.79.227.17:4971 , piece_id=MWEH6NOIWLPPGZXIOJCSK27TQMEZQ26LXEYFK5JBC6TNBM6OTNNQ, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.055530154 (359h11m49.517622637s) [noise=true , address=145.79.227.13:4804 , piece_id=E4NAQCG5VKMWGOKACLWJRCR5CE7U5C2RG5JXSMPZJFV7Z3MUML6Q, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.055825606 (359h11m49.517918089s) [noise=true , address=145.79.227.16:4910 , piece_id=IB26XYX324Z53OHGOVGGXMAWP7DN4O4AJISVKEELZZKCPBSDAOSA, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.055874941 (359h11m49.517967424s) [noise=true , address=145.79.227.18:4005 , piece_id=QKQYWFTEI44E6H3SP7TXWUB3O6XILF3MSK54MZO6TNGINCLPCC2A, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.056729307 (359h11m49.51882179s) [noise=true , address=145.79.227.16:4911 , piece_id=Y5BRGJWOJMK5VXI7VZOH6Q5GO57TQ7MCMDIN274VZUARUKMWE6MQ, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.057750071 (359h11m49.519842594s) [noise=true , address=77.90.31.18:4806 , piece_id=CTR27ZZESJL7RHL5GYAUCCZFJFGEYX4LPV6WVC5A35KCKBW4LLKQ, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.058256756 (359h11m49.520349239s) [noise=true , address=145.79.227.14:4838 , piece_id=CDLY4EVCMBMZNCHC5233J65DAMS7XKT7A5BUHGPNTRLQ7EF2MBPA, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.059035037 (359h11m49.52112752s) [noise=true , address=77.90.31.19:4842 , piece_id=ONWSB7YU5COIBJPUS4ELXDDS43F7GMNXTE75U6CDING4TETZVHSQ, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.059718158 (359h11m49.521810641s) [noise=true , address=45.91.116.9:4262 , piece_id=JFJYRV2LGWGSZXQO4MGZBG7UDJWQD66WC2GW3Z2K6UXZ543SHAMQ, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.064009389 (359h11m49.526101872s) [noise=true , address=45.91.116.6:4975 , piece_id=LWSTSZEWUPVKIXTPGUJCNISF6P24LM324DWNQTOVTGW6C2JFTACQ, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.067343358 (359h11m49.529435841s) [noise=true , address=45.91.116.5:4299 , piece_id=D3WYKJT2QUDYZG4DMDW37GRHR4TALS3R42XOHKC3RL6VCNTR6D2Q, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:05.767894448 (359h11m51.229986921s) [upload=316035]: Next returned nil segment Oct 8 12:43:05.767923374 (359h11m51.230015857s) [upload=316035]: Waiting for error group managing segments... Oct 8 12:43:06.053635749 (359h11m51.515728232s) [noise=true , address=145.79.227.5:4223 , piece_id=5J77YDYAK5W42D6B72C5EHXVUR56LZY6SRTFGF2SO4O3ID42EQPA, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.072509158 (359h11m51.534601641s) [noise=true , address=145.79.227.13:4804 , piece_id=E4NAQCG5VKMWGOKACLWJRCR5CE7U5C2RG5JXSMPZJFV7Z3MUML6Q, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.072607875 (359h11m51.534700368s) [noise=true , address=145.79.227.14:4838 , piece_id=CDLY4EVCMBMZNCHC5233J65DAMS7XKT7A5BUHGPNTRLQ7EF2MBPA, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.073648136 (359h11m51.535740629s) [noise=true , address=145.79.227.17:4971 , piece_id=MWEH6NOIWLPPGZXIOJCSK27TQMEZQ26LXEYFK5JBC6TNBM6OTNNQ, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.123308170 (359h11m51.585400653s) [noise=true , address=145.79.227.18:4005 , piece_id=QKQYWFTEI44E6H3SP7TXWUB3O6XILF3MSK54MZO6TNGINCLPCC2A, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.123445381 (359h11m51.585537864s) [noise=true , address=145.79.227.16:4911 , piece_id=Y5BRGJWOJMK5VXI7VZOH6Q5GO57TQ7MCMDIN274VZUARUKMWE6MQ, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.124920610 (359h11m51.587013093s) [noise=true , address=145.79.227.16:4910 , piece_id=IB26XYX324Z53OHGOVGGXMAWP7DN4O4AJISVKEELZZKCPBSDAOSA, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.229511452 (359h11m51.691603935s) [noise=true , address=77.90.31.17:4770 , piece_id=BD3EFMJ57GUEBFSEOAXPZNBUBFESQ7ONWDVND2ALBCHB6KEOYU7Q, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.263301876 (359h11m51.725394359s) [noise=true , address=45.91.116.9:4262 , piece_id=JFJYRV2LGWGSZXQO4MGZBG7UDJWQD66WC2GW3Z2K6UXZ543SHAMQ, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.273129469 (359h11m51.735221952s) [noise=true , address=77.90.31.19:4842 , piece_id=ONWSB7YU5COIBJPUS4ELXDDS43F7GMNXTE75U6CDING4TETZVHSQ, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.333099946 (359h11m51.795192429s) [noise=true , address=77.90.31.16:4734 , piece_id=TGIPQUVF32I7TKSHMTUM2EK5OD43JVKFOREHL5JTFLFL25LN6QOA, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.342555611 (359h11m51.804648084s) [noise=true , address=45.91.116.5:4299 , piece_id=D3WYKJT2QUDYZG4DMDW37GRHR4TALS3R42XOHKC3RL6VCNTR6D2Q, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.455661779 (359h11m51.917754262s) [noise=true , address=77.90.31.19:4843 , piece_id=IJYZPVKVLJQSJK4WGXNJP2TBXRKYNNLNUEEF6UKGMVWWFSOZ7ZQQ, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.455698799 (359h11m51.917791282s) [seg_pos={0 0}, upload=316035]: Segment reached optimal threshold of 13 pieces. Oct 8 12:43:06.455830440 (359h11m51.917922923s) [noise=true , address=45.91.116.6:4975 , piece_id=LWSTSZEWUPVKIXTPGUJCNISF6P24LM324DWNQTOVTGW6C2JFTACQ, seg_pos={0 0}, upload=316035]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 12:43:06.455841210 (359h11m51.917933693s) [noise=true , address=45.91.116.6:4975 , piece_id=LWSTSZEWUPVKIXTPGUJCNISF6P24LM324DWNQTOVTGW6C2JFTACQ, seg_pos={0 0}, upload=316035]: Long tail cancellation due to optimal threshold Oct 8 12:43:06.455853574 (359h11m51.917946058s) [noise=true , address=77.90.31.21:4878 , piece_id=7LJVVGM6XY7J7XVZKZESOVPRS7ACGYDV7F3QV6TNEIOKA2X6ZEJA, seg_pos={0 0}, upload=316035]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Oct 8 12:43:06.455863152 (359h11m51.917955625s) [noise=true , address=77.90.31.21:4878 , piece_id=7LJVVGM6XY7J7XVZKZESOVPRS7ACGYDV7F3QV6TNEIOKA2X6ZEJA, seg_pos={0 0}, upload=316035]: Long tail cancellation due to optimal threshold Oct 8 12:43:06.455960528 (359h11m51.918053011s) [noise=true , address=77.90.31.18:4806 , piece_id=CTR27ZZESJL7RHL5GYAUCCZFJFGEYX4LPV6WVC5A35KCKBW4LLKQ, seg_pos={0 0}, upload=316035]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:50188->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:50188->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50188->77.90.31.18:4806: use of closed network connection Oct 8 12:43:06.455968313 (359h11m51.918060806s) [noise=true , address=77.90.31.18:4806 , piece_id=CTR27ZZESJL7RHL5GYAUCCZFJFGEYX4LPV6WVC5A35KCKBW4LLKQ, seg_pos={0 0}, upload=316035]: Long tail cancellation due to optimal threshold Oct 8 12:43:06.455985626 (359h11m51.918078129s) [seg_pos={0 0}, upload=316035]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:43:06.456078342 (359h11m51.918170825s) [upload=316035]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:43:06.456084654 (359h11m51.918177137s) [upload=316035]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:43:06.773235514 (359h11m52.235327987s) [upload=316035]: Done uploading segments... Oct 8 12:43:06.773259680 (359h11m52.235352163s) [upload=316035]: Upload finished. err: <nil> Duration: 6.276080618s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.055424874s FAIL (2025-10-08 12:43:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.055455053s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.....XX...................X........X..X..X....XX..XXX.XX..X] duration: 30.003395066s FAIL (2025-10-08 12:43:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: D3WYKJT2QUDYZG4DMDW37GRHR4TALS3R42XOHKC3RL6VCNTR6D2Q) context deadline exceeded; piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: TGIPQUVF32I7TKSHMTUM2EK5OD43JVKFOREHL5JTFLFL25LN6QOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: BD3EFMJ57GUEBFSEOAXPZNBUBFESQ7ONWDVND2ALBCHB6KEOYU7Q) context deadline exceeded ======LOGS====== t 8 12:43:03.793187630 (359h11m49.255280133s) [upload=316035]: Starting upload Oct 8 12:43:03.793285647 (359h11m49.25537813s) [upload=316035]: Done starting upload Oct 8 12:43:03.793485377 (359h11m49.25557786s) [upload=316035]: Uploading segments... Oct 8 12:43:03.793684276 (359h11m49.255776759s) [upload=316035]: Got next segment. Inline: false Oct 8 12:43:03.793763357 (359h11m49.25585584s) [upload=316035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 12:43:03.793770259 (359h11m49.255862733s) [upload=316035]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 12:43:04.052805806 (359h11m49.514898289s) [seg_pos={0 0}, upload=316035]: Begin upload segment... Oct 8 12:43:04.052884667 (359h11m49.51497715s) [seg_pos={0 0}, upload=316035]: Done begin upload segment. Oct 8 12:43:04.052929842 (359h11m49.515022365s) [noise=true , address=77.90.31.16:4734 , piece_id=TGIPQUVF32I7TKSHMTUM2EK5OD43JVKFOREHL5JTFLFL25LN6QOA, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.054231500 (359h11m49.516323984s) [noise=true , address=77.90.31.19:4843 , piece_id=IJYZPVKVLJQSJK4WGXNJP2TBXRKYNNLNUEEF6UKGMVWWFSOZ7ZQQ, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.054375856 (359h11m49.516468339s) [noise=true , address=77.90.31.21:4878 , piece_id=7LJVVGM6XY7J7XVZKZESOVPRS7ACGYDV7F3QV6TNEIOKA2X6ZEJA, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.054786848 (359h11m49.516879331s) [noise=true , address=145.79.227.5:4223 , piece_id=5J77YDYAK5W42D6B72C5EHXVUR56LZY6SRTFGF2SO4O3ID42EQPA, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.055181370 (359h11m49.517273853s) [noise=true , address=77.90.31.17:4770 , piece_id=BD3EFMJ57GUEBFSEOAXPZNBUBFESQ7ONWDVND2ALBCHB6KEOYU7Q, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.055199965 (359h11m49.517292448s) [noise=true , address=145.79.227.17:4971 , piece_id=MWEH6NOIWLPPGZXIOJCSK27TQMEZQ26LXEYFK5JBC6TNBM6OTNNQ, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.055530154 (359h11m49.517622637s) [noise=true , address=145.79.227.13:4804 , piece_id=E4NAQCG5VKMWGOKACLWJRCR5CE7U5C2RG5JXSMPZJFV7Z3MUML6Q, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.055825606 (359h11m49.517918089s) [noise=true , address=145.79.227.16:4910 , piece_id=IB26XYX324Z53OHGOVGGXMAWP7DN4O4AJISVKEELZZKCPBSDAOSA, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.055874941 (359h11m49.517967424s) [noise=true , address=145.79.227.18:4005 , piece_id=QKQYWFTEI44E6H3SP7TXWUB3O6XILF3MSK54MZO6TNGINCLPCC2A, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.056729307 (359h11m49.51882179s) [noise=true , address=145.79.227.16:4911 , piece_id=Y5BRGJWOJMK5VXI7VZOH6Q5GO57TQ7MCMDIN274VZUARUKMWE6MQ, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.057750071 (359h11m49.519842594s) [noise=true , address=77.90.31.18:4806 , piece_id=CTR27ZZESJL7RHL5GYAUCCZFJFGEYX4LPV6WVC5A35KCKBW4LLKQ, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.058256756 (359h11m49.520349239s) [noise=true , address=145.79.227.14:4838 , piece_id=CDLY4EVCMBMZNCHC5233J65DAMS7XKT7A5BUHGPNTRLQ7EF2MBPA, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.059035037 (359h11m49.52112752s) [noise=true , address=77.90.31.19:4842 , piece_id=ONWSB7YU5COIBJPUS4ELXDDS43F7GMNXTE75U6CDING4TETZVHSQ, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.059718158 (359h11m49.521810641s) [noise=true , address=45.91.116.9:4262 , piece_id=JFJYRV2LGWGSZXQO4MGZBG7UDJWQD66WC2GW3Z2K6UXZ543SHAMQ, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.064009389 (359h11m49.526101872s) [noise=true , address=45.91.116.6:4975 , piece_id=LWSTSZEWUPVKIXTPGUJCNISF6P24LM324DWNQTOVTGW6C2JFTACQ, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:04.067343358 (359h11m49.529435841s) [noise=true , address=45.91.116.5:4299 , piece_id=D3WYKJT2QUDYZG4DMDW37GRHR4TALS3R42XOHKC3RL6VCNTR6D2Q, seg_pos={0 0}, upload=316035]: Uploading piece... Oct 8 12:43:05.767894448 (359h11m51.229986921s) [upload=316035]: Next returned nil segment Oct 8 12:43:05.767923374 (359h11m51.230015857s) [upload=316035]: Waiting for error group managing segments... Oct 8 12:43:06.053635749 (359h11m51.515728232s) [noise=true , address=145.79.227.5:4223 , piece_id=5J77YDYAK5W42D6B72C5EHXVUR56LZY6SRTFGF2SO4O3ID42EQPA, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.072509158 (359h11m51.534601641s) [noise=true , address=145.79.227.13:4804 , piece_id=E4NAQCG5VKMWGOKACLWJRCR5CE7U5C2RG5JXSMPZJFV7Z3MUML6Q, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.072607875 (359h11m51.534700368s) [noise=true , address=145.79.227.14:4838 , piece_id=CDLY4EVCMBMZNCHC5233J65DAMS7XKT7A5BUHGPNTRLQ7EF2MBPA, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.073648136 (359h11m51.535740629s) [noise=true , address=145.79.227.17:4971 , piece_id=MWEH6NOIWLPPGZXIOJCSK27TQMEZQ26LXEYFK5JBC6TNBM6OTNNQ, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.123308170 (359h11m51.585400653s) [noise=true , address=145.79.227.18:4005 , piece_id=QKQYWFTEI44E6H3SP7TXWUB3O6XILF3MSK54MZO6TNGINCLPCC2A, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.123445381 (359h11m51.585537864s) [noise=true , address=145.79.227.16:4911 , piece_id=Y5BRGJWOJMK5VXI7VZOH6Q5GO57TQ7MCMDIN274VZUARUKMWE6MQ, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.124920610 (359h11m51.587013093s) [noise=true , address=145.79.227.16:4910 , piece_id=IB26XYX324Z53OHGOVGGXMAWP7DN4O4AJISVKEELZZKCPBSDAOSA, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.229511452 (359h11m51.691603935s) [noise=true , address=77.90.31.17:4770 , piece_id=BD3EFMJ57GUEBFSEOAXPZNBUBFESQ7ONWDVND2ALBCHB6KEOYU7Q, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.263301876 (359h11m51.725394359s) [noise=true , address=45.91.116.9:4262 , piece_id=JFJYRV2LGWGSZXQO4MGZBG7UDJWQD66WC2GW3Z2K6UXZ543SHAMQ, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.273129469 (359h11m51.735221952s) [noise=true , address=77.90.31.19:4842 , piece_id=ONWSB7YU5COIBJPUS4ELXDDS43F7GMNXTE75U6CDING4TETZVHSQ, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.333099946 (359h11m51.795192429s) [noise=true , address=77.90.31.16:4734 , piece_id=TGIPQUVF32I7TKSHMTUM2EK5OD43JVKFOREHL5JTFLFL25LN6QOA, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.342555611 (359h11m51.804648084s) [noise=true , address=45.91.116.5:4299 , piece_id=D3WYKJT2QUDYZG4DMDW37GRHR4TALS3R42XOHKC3RL6VCNTR6D2Q, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.455661779 (359h11m51.917754262s) [noise=true , address=77.90.31.19:4843 , piece_id=IJYZPVKVLJQSJK4WGXNJP2TBXRKYNNLNUEEF6UKGMVWWFSOZ7ZQQ, seg_pos={0 0}, upload=316035]: Done uploading piece. err: <nil> Oct 8 12:43:06.455698799 (359h11m51.917791282s) [seg_pos={0 0}, upload=316035]: Segment reached optimal threshold of 13 pieces. Oct 8 12:43:06.455830440 (359h11m51.917922923s) [noise=true , address=45.91.116.6:4975 , piece_id=LWSTSZEWUPVKIXTPGUJCNISF6P24LM324DWNQTOVTGW6C2JFTACQ, seg_pos={0 0}, upload=316035]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 12:43:06.455841210 (359h11m51.917933693s) [noise=true , address=45.91.116.6:4975 , piece_id=LWSTSZEWUPVKIXTPGUJCNISF6P24LM324DWNQTOVTGW6C2JFTACQ, seg_pos={0 0}, upload=316035]: Long tail cancellation due to optimal threshold Oct 8 12:43:06.455853574 (359h11m51.917946058s) [noise=true , address=77.90.31.21:4878 , piece_id=7LJVVGM6XY7J7XVZKZESOVPRS7ACGYDV7F3QV6TNEIOKA2X6ZEJA, seg_pos={0 0}, upload=316035]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Oct 8 12:43:06.455863152 (359h11m51.917955625s) [noise=true , address=77.90.31.21:4878 , piece_id=7LJVVGM6XY7J7XVZKZESOVPRS7ACGYDV7F3QV6TNEIOKA2X6ZEJA, seg_pos={0 0}, upload=316035]: Long tail cancellation due to optimal threshold Oct 8 12:43:06.455960528 (359h11m51.918053011s) [noise=true , address=77.90.31.18:4806 , piece_id=CTR27ZZESJL7RHL5GYAUCCZFJFGEYX4LPV6WVC5A35KCKBW4LLKQ, seg_pos={0 0}, upload=316035]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:50188->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:50188->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50188->77.90.31.18:4806: use of closed network connection Oct 8 12:43:06.455968313 (359h11m51.918060806s) [noise=true , address=77.90.31.18:4806 , piece_id=CTR27ZZESJL7RHL5GYAUCCZFJFGEYX4LPV6WVC5A35KCKBW4LLKQ, seg_pos={0 0}, upload=316035]: Long tail cancellation due to optimal threshold Oct 8 12:43:06.455985626 (359h11m51.918078129s) [seg_pos={0 0}, upload=316035]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 12:43:06.456078342 (359h11m51.918170825s) [upload=316035]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 12:43:06.456084654 (359h11m51.918177137s) [upload=316035]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 12:43:06.773235514 (359h11m52.235327987s) [upload=316035]: Done uploading segments... Oct 8 12:43:06.773259680 (359h11m52.235352163s) [upload=316035]: Upload finished. err: <nil> Duration: 30.003424611s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)