Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................X........................X............] duration:
5.839569201s
FAIL (2025-07-09 01:32:48 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ommit
Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments...
Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil>
Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload
Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload
Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments...
Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false
Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment...
Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment.
Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment
Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments...
Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection
Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection
Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection
Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments...
Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.839593187s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [....X.....X...X......XX....X...X.X....X...X.X..X...X..X.....] duration:
5.955888364s
FAIL (2025-07-09 01:32:34 UTC)
orbiter warning: context deadline exceeded
Duration: 5.955907521s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.............................X..........................X...] duration:
32.627569303s
FAIL (2025-07-09 01:32:24 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
RequestItem_SegmentBegin
Jul 9 01:29:01.337433896 (376h18m36.108075261s) [seg_pos={0 0}, upload=301063]: Begin upload segment...
Jul 9 01:29:01.337487208 (376h18m36.108128543s) [seg_pos={0 0}, upload=301063]: Done begin upload segment.
Jul 9 01:29:01.337511635 (376h18m36.10815298s) [noise=true , address=77.90.31.11:4563 , piece_id=BC47C6JJI2QGZT54RFSVTB34IWFZVY2MYP4R55HYM3MC6PZ63L3Q, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.337520201 (376h18m36.108161546s) [noise=true , address=104.36.82.35:4614 , piece_id=JTXUCFS36ZNLCZHAEZORW77F3ZNAN5M6FYKQ2ZFYUAFXY5DRC3GQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.338221482 (376h18m36.108862827s) [noise=true , address=77.90.31.21:4909 , piece_id=BDRGEIFZ56SQ46453YLMKGZZE5F4FHIUJE35RU5RCUNDNQFXYSPQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.338235780 (376h18m36.108877145s) [noise=true , address=77.90.31.10:4533 , piece_id=CMUMWPDGLP2XWKFHCSXNE3IB7HYJEDOVYQNUM7DKI7THVBIHNDHQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.338701732 (376h18m36.109343067s) [noise=true , address=77.90.31.15:4701 , piece_id=M6US4USD57GPEL35W4IJF73X3BJMDRBLD2UXSHTLTYHMVI6XCTTA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.338733382 (376h18m36.109374717s) [noise=true , address=77.90.31.14:4682 , piece_id=KWYECXGPZ4TVAXVUK5IJHYRW36OFZGXFRWDLUSPJNZUUO5JDGLZA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339087590 (376h18m36.109728925s) [noise=true , address=77.90.31.17:4786 , piece_id=X4VZDLZC2AAXL5VRABJE5PIU6RJGJOQH67VHMSLT4VPLMF64QQ2A, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339094813 (376h18m36.109736159s) [noise=true , address=77.90.31.13:4651 , piece_id=TF3NGJ4AASQSXY4JSF27FCW7VSHF5VWYLHMNXHEQAMT5KJWXVNQQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339465213 (376h18m36.110106558s) [noise=true , address=77.90.31.12:4606 , piece_id=JCQCFDBA4YSIFSKS4RFYFUUYUAFU3XR2KI7VV4FCGFD4POFGXCHA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339475853 (376h18m36.110117198s) [noise=true , address=104.36.82.18:4391 , piece_id=FI7VKWZELHQFI2G4R5GOPAM4NLRT4VKGBJ26POQ2IS24F5EW77ZA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339842684 (376h18m36.110484019s) [noise=true , address=104.36.82.5:4868 , piece_id=W7Q3QS3IKB7X6TOOSKKMX5W42BHG274PXYNZ7UZCGSPNQXUFAURQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339849308 (376h18m36.110490643s) [noise=true , address=77.90.31.10:4537 , piece_id=WQKTKYRY4C32EGPMHSXQAS3KSRAWIDDF7BYBZ2MKZOWHKFEXBGLA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.340181744 (376h18m36.110823079s) [noise=true , address=104.36.82.36:4651 , piece_id=QIMZ4CWZJCOYIIWEZGQYWZ2EHHL7DYAHHQ2YX5W7PPCH4QJNWS4A, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.340191722 (376h18m36.110833057s) [noise=true , address=104.36.82.19:4433 , piece_id=VUXOICADHIDKYIDNKJXDN3WA6FNDG3XJR3OCXC4UC26EA5P3NIMA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.340777823 (376h18m36.111419158s) [noise=true , address=77.90.31.11:4581 , piece_id=TKMT65PF3VOTCHFY6GBFC2W5JYOJTAHK5WFRZF3YAPOYHA7SGFDQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.341080963 (376h18m36.111722308s) [noise=true , address=104.36.82.44:4880 , piece_id=44M5JZ2MTKYBK6TH5HX46KAL24UCXIO4X4PV6DY6VUE4JHGKJXRA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.342101286 (376h18m36.112742631s) [noise=true , address=104.36.82.21:4477 , piece_id=N4BEJUNAM2RKLEZ7A7XKJI525EBWQ35UBMMSBZLIIC5AXNLMXCKA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.344027331 (376h18m36.114668667s) [noise=true , address=77.90.31.16:4757 , piece_id=HUJGTDFFEA5ATNB2HXCBERZYDL5N5BLMZYW6G3DASQTC2PM4IFYA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:02.921702307 (376h18m37.692343642s) [upload=301063]: Next returned nil segment
Jul 9 01:29:02.921717205 (376h18m37.69235854s) [upload=301063]: Waiting for error group managing segments...
Jul 9 01:29:03.330770575 (376h18m38.10141191s) [noise=true , address=77.90.31.11:4581 , piece_id=TKMT65PF3VOTCHFY6GBFC2W5JYOJTAHK5WFRZF3YAPOYHA7SGFDQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.333021060 (376h18m38.103662395s) [noise=true , address=77.90.31.15:4701 , piece_id=M6US4USD57GPEL35W4IJF73X3BJMDRBLD2UXSHTLTYHMVI6XCTTA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.346762402 (376h18m38.117403737s) [noise=true , address=77.90.31.14:4682 , piece_id=KWYECXGPZ4TVAXVUK5IJHYRW36OFZGXFRWDLUSPJNZUUO5JDGLZA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.346815323 (376h18m38.117456658s) [noise=true , address=77.90.31.11:4563 , piece_id=BC47C6JJI2QGZT54RFSVTB34IWFZVY2MYP4R55HYM3MC6PZ63L3Q, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.347060412 (376h18m38.117701747s) [noise=true , address=77.90.31.17:4786 , piece_id=X4VZDLZC2AAXL5VRABJE5PIU6RJGJOQH67VHMSLT4VPLMF64QQ2A, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.347185391 (376h18m38.117826726s) [noise=true , address=77.90.31.21:4909 , piece_id=BDRGEIFZ56SQ46453YLMKGZZE5F4FHIUJE35RU5RCUNDNQFXYSPQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.347425851 (376h18m38.118067186s) [noise=true , address=77.90.31.13:4651 , piece_id=TF3NGJ4AASQSXY4JSF27FCW7VSHF5VWYLHMNXHEQAMT5KJWXVNQQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.347529449 (376h18m38.118170784s) [noise=true , address=77.90.31.12:4606 , piece_id=JCQCFDBA4YSIFSKS4RFYFUUYUAFU3XR2KI7VV4FCGFD4POFGXCHA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.347923393 (376h18m38.118564728s) [noise=true , address=77.90.31.10:4537 , piece_id=WQKTKYRY4C32EGPMHSXQAS3KSRAWIDDF7BYBZ2MKZOWHKFEXBGLA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.348515206 (376h18m38.119156541s) [noise=true , address=77.90.31.16:4757 , piece_id=HUJGTDFFEA5ATNB2HXCBERZYDL5N5BLMZYW6G3DASQTC2PM4IFYA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.363502611 (376h18m38.134143946s) [noise=true , address=77.90.31.10:4533 , piece_id=CMUMWPDGLP2XWKFHCSXNE3IB7HYJEDOVYQNUM7DKI7THVBIHNDHQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:26.444658884 (376h19m1.215300209s) [noise=true , address=104.36.82.19:4433 , piece_id=VUXOICADHIDKYIDNKJXDN3WA6FNDG3XJR3OCXC4UC26EA5P3NIMA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:37.476126743 (376h19m12.246768078s) [noise=true , address=104.36.82.36:4651 , piece_id=QIMZ4CWZJCOYIIWEZGQYWZ2EHHL7DYAHHQ2YX5W7PPCH4QJNWS4A, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:56.449247029 (376h19m31.219888364s) [noise=true , address=104.36.82.5:4868 , piece_id=W7Q3QS3IKB7X6TOOSKKMX5W42BHG274PXYNZ7UZCGSPNQXUFAURQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:56.449272748 (376h19m31.219914073s) [seg_pos={0 0}, upload=301063]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:29:56.449354975 (376h19m31.2199963s) [noise=true , address=104.36.82.21:4477 , piece_id=N4BEJUNAM2RKLEZ7A7XKJI525EBWQ35UBMMSBZLIIC5AXNLMXCKA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF
Jul 9 01:29:56.449457422 (376h19m31.220098757s) [noise=true , address=104.36.82.44:4880 , piece_id=44M5JZ2MTKYBK6TH5HX46KAL24UCXIO4X4PV6DY6VUE4JHGKJXRA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Jul 9 01:29:56.449486968 (376h19m31.220128303s) [noise=true , address=104.36.82.18:4391 , piece_id=FI7VKWZELHQFI2G4R5GOPAM4NLRT4VKGBJ26POQ2IS24F5EW77ZA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MHmmjheT4VDX93MiQNzQyTWkxZFzz2PwNvdRfsevZRbj6qmJS): protocol: write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection; write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection
Jul 9 01:29:56.449513970 (376h19m31.220155295s) [noise=true , address=104.36.82.35:4614 , piece_id=JTXUCFS36ZNLCZHAEZORW77F3ZNAN5M6FYKQ2ZFYUAFXY5DRC3GQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Pua5x8rAsuqYWCgoAY6fgy2SXNoDjzt7WwgMhzv1HN2KG5NY4): protocol: write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection; write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection
Jul 9 01:29:56.449528637 (376h19m31.220169972s) [seg_pos={0 0}, upload=301063]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:29:56.449590266 (376h19m31.220231601s) [upload=301063]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:29:56.449593853 (376h19m31.220235178s) [upload=301063]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:29:56.719973407 (376h19m31.490614742s) [upload=301063]: Done uploading segments...
Jul 9 01:29:56.719991001 (376h19m31.490632336s) [upload=301063]: Upload finished. err: <nil>
Duration: 32.627594772s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.......................................X......XX.XX..X......] duration:
40.798932153s
FAIL (2025-07-09 01:32:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 40.798953113s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................X........................X...........] duration:
5.754527734s
FAIL (2025-07-09 01:31:53 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ommit
Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments...
Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil>
Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload
Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload
Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments...
Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false
Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment...
Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment.
Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment
Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments...
Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection
Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection
Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection
Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments...
Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.754553233s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [X....X.....X...X......XX....X...X.X....X...X.X..X...X..X....] duration:
27.593250612s
FAIL (2025-07-09 01:31:53 UTC)
orbiter warning: context deadline exceeded
Duration: 27.593271111s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..............................X..........................X..] duration:
37.244516331s
FAIL (2025-07-09 01:31:37 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
RequestItem_SegmentBegin
Jul 9 01:29:01.337433896 (376h18m36.108075261s) [seg_pos={0 0}, upload=301063]: Begin upload segment...
Jul 9 01:29:01.337487208 (376h18m36.108128543s) [seg_pos={0 0}, upload=301063]: Done begin upload segment.
Jul 9 01:29:01.337511635 (376h18m36.10815298s) [noise=true , address=77.90.31.11:4563 , piece_id=BC47C6JJI2QGZT54RFSVTB34IWFZVY2MYP4R55HYM3MC6PZ63L3Q, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.337520201 (376h18m36.108161546s) [noise=true , address=104.36.82.35:4614 , piece_id=JTXUCFS36ZNLCZHAEZORW77F3ZNAN5M6FYKQ2ZFYUAFXY5DRC3GQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.338221482 (376h18m36.108862827s) [noise=true , address=77.90.31.21:4909 , piece_id=BDRGEIFZ56SQ46453YLMKGZZE5F4FHIUJE35RU5RCUNDNQFXYSPQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.338235780 (376h18m36.108877145s) [noise=true , address=77.90.31.10:4533 , piece_id=CMUMWPDGLP2XWKFHCSXNE3IB7HYJEDOVYQNUM7DKI7THVBIHNDHQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.338701732 (376h18m36.109343067s) [noise=true , address=77.90.31.15:4701 , piece_id=M6US4USD57GPEL35W4IJF73X3BJMDRBLD2UXSHTLTYHMVI6XCTTA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.338733382 (376h18m36.109374717s) [noise=true , address=77.90.31.14:4682 , piece_id=KWYECXGPZ4TVAXVUK5IJHYRW36OFZGXFRWDLUSPJNZUUO5JDGLZA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339087590 (376h18m36.109728925s) [noise=true , address=77.90.31.17:4786 , piece_id=X4VZDLZC2AAXL5VRABJE5PIU6RJGJOQH67VHMSLT4VPLMF64QQ2A, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339094813 (376h18m36.109736159s) [noise=true , address=77.90.31.13:4651 , piece_id=TF3NGJ4AASQSXY4JSF27FCW7VSHF5VWYLHMNXHEQAMT5KJWXVNQQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339465213 (376h18m36.110106558s) [noise=true , address=77.90.31.12:4606 , piece_id=JCQCFDBA4YSIFSKS4RFYFUUYUAFU3XR2KI7VV4FCGFD4POFGXCHA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339475853 (376h18m36.110117198s) [noise=true , address=104.36.82.18:4391 , piece_id=FI7VKWZELHQFI2G4R5GOPAM4NLRT4VKGBJ26POQ2IS24F5EW77ZA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339842684 (376h18m36.110484019s) [noise=true , address=104.36.82.5:4868 , piece_id=W7Q3QS3IKB7X6TOOSKKMX5W42BHG274PXYNZ7UZCGSPNQXUFAURQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339849308 (376h18m36.110490643s) [noise=true , address=77.90.31.10:4537 , piece_id=WQKTKYRY4C32EGPMHSXQAS3KSRAWIDDF7BYBZ2MKZOWHKFEXBGLA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.340181744 (376h18m36.110823079s) [noise=true , address=104.36.82.36:4651 , piece_id=QIMZ4CWZJCOYIIWEZGQYWZ2EHHL7DYAHHQ2YX5W7PPCH4QJNWS4A, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.340191722 (376h18m36.110833057s) [noise=true , address=104.36.82.19:4433 , piece_id=VUXOICADHIDKYIDNKJXDN3WA6FNDG3XJR3OCXC4UC26EA5P3NIMA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.340777823 (376h18m36.111419158s) [noise=true , address=77.90.31.11:4581 , piece_id=TKMT65PF3VOTCHFY6GBFC2W5JYOJTAHK5WFRZF3YAPOYHA7SGFDQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.341080963 (376h18m36.111722308s) [noise=true , address=104.36.82.44:4880 , piece_id=44M5JZ2MTKYBK6TH5HX46KAL24UCXIO4X4PV6DY6VUE4JHGKJXRA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.342101286 (376h18m36.112742631s) [noise=true , address=104.36.82.21:4477 , piece_id=N4BEJUNAM2RKLEZ7A7XKJI525EBWQ35UBMMSBZLIIC5AXNLMXCKA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.344027331 (376h18m36.114668667s) [noise=true , address=77.90.31.16:4757 , piece_id=HUJGTDFFEA5ATNB2HXCBERZYDL5N5BLMZYW6G3DASQTC2PM4IFYA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:02.921702307 (376h18m37.692343642s) [upload=301063]: Next returned nil segment
Jul 9 01:29:02.921717205 (376h18m37.69235854s) [upload=301063]: Waiting for error group managing segments...
Jul 9 01:29:03.330770575 (376h18m38.10141191s) [noise=true , address=77.90.31.11:4581 , piece_id=TKMT65PF3VOTCHFY6GBFC2W5JYOJTAHK5WFRZF3YAPOYHA7SGFDQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.333021060 (376h18m38.103662395s) [noise=true , address=77.90.31.15:4701 , piece_id=M6US4USD57GPEL35W4IJF73X3BJMDRBLD2UXSHTLTYHMVI6XCTTA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.346762402 (376h18m38.117403737s) [noise=true , address=77.90.31.14:4682 , piece_id=KWYECXGPZ4TVAXVUK5IJHYRW36OFZGXFRWDLUSPJNZUUO5JDGLZA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.346815323 (376h18m38.117456658s) [noise=true , address=77.90.31.11:4563 , piece_id=BC47C6JJI2QGZT54RFSVTB34IWFZVY2MYP4R55HYM3MC6PZ63L3Q, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.347060412 (376h18m38.117701747s) [noise=true , address=77.90.31.17:4786 , piece_id=X4VZDLZC2AAXL5VRABJE5PIU6RJGJOQH67VHMSLT4VPLMF64QQ2A, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.347185391 (376h18m38.117826726s) [noise=true , address=77.90.31.21:4909 , piece_id=BDRGEIFZ56SQ46453YLMKGZZE5F4FHIUJE35RU5RCUNDNQFXYSPQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.347425851 (376h18m38.118067186s) [noise=true , address=77.90.31.13:4651 , piece_id=TF3NGJ4AASQSXY4JSF27FCW7VSHF5VWYLHMNXHEQAMT5KJWXVNQQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.347529449 (376h18m38.118170784s) [noise=true , address=77.90.31.12:4606 , piece_id=JCQCFDBA4YSIFSKS4RFYFUUYUAFU3XR2KI7VV4FCGFD4POFGXCHA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.347923393 (376h18m38.118564728s) [noise=true , address=77.90.31.10:4537 , piece_id=WQKTKYRY4C32EGPMHSXQAS3KSRAWIDDF7BYBZ2MKZOWHKFEXBGLA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.348515206 (376h18m38.119156541s) [noise=true , address=77.90.31.16:4757 , piece_id=HUJGTDFFEA5ATNB2HXCBERZYDL5N5BLMZYW6G3DASQTC2PM4IFYA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.363502611 (376h18m38.134143946s) [noise=true , address=77.90.31.10:4533 , piece_id=CMUMWPDGLP2XWKFHCSXNE3IB7HYJEDOVYQNUM7DKI7THVBIHNDHQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:26.444658884 (376h19m1.215300209s) [noise=true , address=104.36.82.19:4433 , piece_id=VUXOICADHIDKYIDNKJXDN3WA6FNDG3XJR3OCXC4UC26EA5P3NIMA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:37.476126743 (376h19m12.246768078s) [noise=true , address=104.36.82.36:4651 , piece_id=QIMZ4CWZJCOYIIWEZGQYWZ2EHHL7DYAHHQ2YX5W7PPCH4QJNWS4A, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:56.449247029 (376h19m31.219888364s) [noise=true , address=104.36.82.5:4868 , piece_id=W7Q3QS3IKB7X6TOOSKKMX5W42BHG274PXYNZ7UZCGSPNQXUFAURQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:56.449272748 (376h19m31.219914073s) [seg_pos={0 0}, upload=301063]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:29:56.449354975 (376h19m31.2199963s) [noise=true , address=104.36.82.21:4477 , piece_id=N4BEJUNAM2RKLEZ7A7XKJI525EBWQ35UBMMSBZLIIC5AXNLMXCKA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF
Jul 9 01:29:56.449457422 (376h19m31.220098757s) [noise=true , address=104.36.82.44:4880 , piece_id=44M5JZ2MTKYBK6TH5HX46KAL24UCXIO4X4PV6DY6VUE4JHGKJXRA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Jul 9 01:29:56.449486968 (376h19m31.220128303s) [noise=true , address=104.36.82.18:4391 , piece_id=FI7VKWZELHQFI2G4R5GOPAM4NLRT4VKGBJ26POQ2IS24F5EW77ZA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MHmmjheT4VDX93MiQNzQyTWkxZFzz2PwNvdRfsevZRbj6qmJS): protocol: write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection; write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection
Jul 9 01:29:56.449513970 (376h19m31.220155295s) [noise=true , address=104.36.82.35:4614 , piece_id=JTXUCFS36ZNLCZHAEZORW77F3ZNAN5M6FYKQ2ZFYUAFXY5DRC3GQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Pua5x8rAsuqYWCgoAY6fgy2SXNoDjzt7WwgMhzv1HN2KG5NY4): protocol: write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection; write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection
Jul 9 01:29:56.449528637 (376h19m31.220169972s) [seg_pos={0 0}, upload=301063]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:29:56.449590266 (376h19m31.220231601s) [upload=301063]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:29:56.449593853 (376h19m31.220235178s) [upload=301063]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:29:56.719973407 (376h19m31.490614742s) [upload=301063]: Done uploading segments...
Jul 9 01:29:56.719991001 (376h19m31.490632336s) [upload=301063]: Upload finished. err: <nil>
Duration: 37.244540317s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................X........................X..........] duration:
20.091453485s
FAIL (2025-07-09 01:31:12 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ommit
Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments...
Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil>
Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload
Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload
Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments...
Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false
Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment...
Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment.
Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment
Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments...
Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection
Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection
Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection
Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments...
Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 20.091485286s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [........................................X......XX.XX..X.....] duration:
43.633687478s
FAIL (2025-07-09 01:31:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 43.633719088s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [XX....X.....X...X......XX....X...X.X....X...X.X..X...X..X...] duration:
12.901935194s
FAIL (2025-07-09 01:30:41 UTC)
orbiter warning: context deadline exceeded
Duration: 12.901981453s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...............................X..........................X.] duration:
26.508052435s
FAIL (2025-07-09 01:30:39 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
RequestItem_SegmentBegin
Jul 9 01:29:01.337433896 (376h18m36.108075261s) [seg_pos={0 0}, upload=301063]: Begin upload segment...
Jul 9 01:29:01.337487208 (376h18m36.108128543s) [seg_pos={0 0}, upload=301063]: Done begin upload segment.
Jul 9 01:29:01.337511635 (376h18m36.10815298s) [noise=true , address=77.90.31.11:4563 , piece_id=BC47C6JJI2QGZT54RFSVTB34IWFZVY2MYP4R55HYM3MC6PZ63L3Q, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.337520201 (376h18m36.108161546s) [noise=true , address=104.36.82.35:4614 , piece_id=JTXUCFS36ZNLCZHAEZORW77F3ZNAN5M6FYKQ2ZFYUAFXY5DRC3GQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.338221482 (376h18m36.108862827s) [noise=true , address=77.90.31.21:4909 , piece_id=BDRGEIFZ56SQ46453YLMKGZZE5F4FHIUJE35RU5RCUNDNQFXYSPQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.338235780 (376h18m36.108877145s) [noise=true , address=77.90.31.10:4533 , piece_id=CMUMWPDGLP2XWKFHCSXNE3IB7HYJEDOVYQNUM7DKI7THVBIHNDHQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.338701732 (376h18m36.109343067s) [noise=true , address=77.90.31.15:4701 , piece_id=M6US4USD57GPEL35W4IJF73X3BJMDRBLD2UXSHTLTYHMVI6XCTTA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.338733382 (376h18m36.109374717s) [noise=true , address=77.90.31.14:4682 , piece_id=KWYECXGPZ4TVAXVUK5IJHYRW36OFZGXFRWDLUSPJNZUUO5JDGLZA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339087590 (376h18m36.109728925s) [noise=true , address=77.90.31.17:4786 , piece_id=X4VZDLZC2AAXL5VRABJE5PIU6RJGJOQH67VHMSLT4VPLMF64QQ2A, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339094813 (376h18m36.109736159s) [noise=true , address=77.90.31.13:4651 , piece_id=TF3NGJ4AASQSXY4JSF27FCW7VSHF5VWYLHMNXHEQAMT5KJWXVNQQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339465213 (376h18m36.110106558s) [noise=true , address=77.90.31.12:4606 , piece_id=JCQCFDBA4YSIFSKS4RFYFUUYUAFU3XR2KI7VV4FCGFD4POFGXCHA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339475853 (376h18m36.110117198s) [noise=true , address=104.36.82.18:4391 , piece_id=FI7VKWZELHQFI2G4R5GOPAM4NLRT4VKGBJ26POQ2IS24F5EW77ZA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339842684 (376h18m36.110484019s) [noise=true , address=104.36.82.5:4868 , piece_id=W7Q3QS3IKB7X6TOOSKKMX5W42BHG274PXYNZ7UZCGSPNQXUFAURQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339849308 (376h18m36.110490643s) [noise=true , address=77.90.31.10:4537 , piece_id=WQKTKYRY4C32EGPMHSXQAS3KSRAWIDDF7BYBZ2MKZOWHKFEXBGLA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.340181744 (376h18m36.110823079s) [noise=true , address=104.36.82.36:4651 , piece_id=QIMZ4CWZJCOYIIWEZGQYWZ2EHHL7DYAHHQ2YX5W7PPCH4QJNWS4A, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.340191722 (376h18m36.110833057s) [noise=true , address=104.36.82.19:4433 , piece_id=VUXOICADHIDKYIDNKJXDN3WA6FNDG3XJR3OCXC4UC26EA5P3NIMA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.340777823 (376h18m36.111419158s) [noise=true , address=77.90.31.11:4581 , piece_id=TKMT65PF3VOTCHFY6GBFC2W5JYOJTAHK5WFRZF3YAPOYHA7SGFDQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.341080963 (376h18m36.111722308s) [noise=true , address=104.36.82.44:4880 , piece_id=44M5JZ2MTKYBK6TH5HX46KAL24UCXIO4X4PV6DY6VUE4JHGKJXRA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.342101286 (376h18m36.112742631s) [noise=true , address=104.36.82.21:4477 , piece_id=N4BEJUNAM2RKLEZ7A7XKJI525EBWQ35UBMMSBZLIIC5AXNLMXCKA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.344027331 (376h18m36.114668667s) [noise=true , address=77.90.31.16:4757 , piece_id=HUJGTDFFEA5ATNB2HXCBERZYDL5N5BLMZYW6G3DASQTC2PM4IFYA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:02.921702307 (376h18m37.692343642s) [upload=301063]: Next returned nil segment
Jul 9 01:29:02.921717205 (376h18m37.69235854s) [upload=301063]: Waiting for error group managing segments...
Jul 9 01:29:03.330770575 (376h18m38.10141191s) [noise=true , address=77.90.31.11:4581 , piece_id=TKMT65PF3VOTCHFY6GBFC2W5JYOJTAHK5WFRZF3YAPOYHA7SGFDQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.333021060 (376h18m38.103662395s) [noise=true , address=77.90.31.15:4701 , piece_id=M6US4USD57GPEL35W4IJF73X3BJMDRBLD2UXSHTLTYHMVI6XCTTA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.346762402 (376h18m38.117403737s) [noise=true , address=77.90.31.14:4682 , piece_id=KWYECXGPZ4TVAXVUK5IJHYRW36OFZGXFRWDLUSPJNZUUO5JDGLZA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.346815323 (376h18m38.117456658s) [noise=true , address=77.90.31.11:4563 , piece_id=BC47C6JJI2QGZT54RFSVTB34IWFZVY2MYP4R55HYM3MC6PZ63L3Q, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.347060412 (376h18m38.117701747s) [noise=true , address=77.90.31.17:4786 , piece_id=X4VZDLZC2AAXL5VRABJE5PIU6RJGJOQH67VHMSLT4VPLMF64QQ2A, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.347185391 (376h18m38.117826726s) [noise=true , address=77.90.31.21:4909 , piece_id=BDRGEIFZ56SQ46453YLMKGZZE5F4FHIUJE35RU5RCUNDNQFXYSPQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.347425851 (376h18m38.118067186s) [noise=true , address=77.90.31.13:4651 , piece_id=TF3NGJ4AASQSXY4JSF27FCW7VSHF5VWYLHMNXHEQAMT5KJWXVNQQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.347529449 (376h18m38.118170784s) [noise=true , address=77.90.31.12:4606 , piece_id=JCQCFDBA4YSIFSKS4RFYFUUYUAFU3XR2KI7VV4FCGFD4POFGXCHA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.347923393 (376h18m38.118564728s) [noise=true , address=77.90.31.10:4537 , piece_id=WQKTKYRY4C32EGPMHSXQAS3KSRAWIDDF7BYBZ2MKZOWHKFEXBGLA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.348515206 (376h18m38.119156541s) [noise=true , address=77.90.31.16:4757 , piece_id=HUJGTDFFEA5ATNB2HXCBERZYDL5N5BLMZYW6G3DASQTC2PM4IFYA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.363502611 (376h18m38.134143946s) [noise=true , address=77.90.31.10:4533 , piece_id=CMUMWPDGLP2XWKFHCSXNE3IB7HYJEDOVYQNUM7DKI7THVBIHNDHQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:26.444658884 (376h19m1.215300209s) [noise=true , address=104.36.82.19:4433 , piece_id=VUXOICADHIDKYIDNKJXDN3WA6FNDG3XJR3OCXC4UC26EA5P3NIMA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:37.476126743 (376h19m12.246768078s) [noise=true , address=104.36.82.36:4651 , piece_id=QIMZ4CWZJCOYIIWEZGQYWZ2EHHL7DYAHHQ2YX5W7PPCH4QJNWS4A, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:56.449247029 (376h19m31.219888364s) [noise=true , address=104.36.82.5:4868 , piece_id=W7Q3QS3IKB7X6TOOSKKMX5W42BHG274PXYNZ7UZCGSPNQXUFAURQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:56.449272748 (376h19m31.219914073s) [seg_pos={0 0}, upload=301063]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:29:56.449354975 (376h19m31.2199963s) [noise=true , address=104.36.82.21:4477 , piece_id=N4BEJUNAM2RKLEZ7A7XKJI525EBWQ35UBMMSBZLIIC5AXNLMXCKA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF
Jul 9 01:29:56.449457422 (376h19m31.220098757s) [noise=true , address=104.36.82.44:4880 , piece_id=44M5JZ2MTKYBK6TH5HX46KAL24UCXIO4X4PV6DY6VUE4JHGKJXRA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Jul 9 01:29:56.449486968 (376h19m31.220128303s) [noise=true , address=104.36.82.18:4391 , piece_id=FI7VKWZELHQFI2G4R5GOPAM4NLRT4VKGBJ26POQ2IS24F5EW77ZA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MHmmjheT4VDX93MiQNzQyTWkxZFzz2PwNvdRfsevZRbj6qmJS): protocol: write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection; write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection
Jul 9 01:29:56.449513970 (376h19m31.220155295s) [noise=true , address=104.36.82.35:4614 , piece_id=JTXUCFS36ZNLCZHAEZORW77F3ZNAN5M6FYKQ2ZFYUAFXY5DRC3GQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Pua5x8rAsuqYWCgoAY6fgy2SXNoDjzt7WwgMhzv1HN2KG5NY4): protocol: write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection; write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection
Jul 9 01:29:56.449528637 (376h19m31.220169972s) [seg_pos={0 0}, upload=301063]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:29:56.449590266 (376h19m31.220231601s) [upload=301063]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:29:56.449593853 (376h19m31.220235178s) [upload=301063]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:29:56.719973407 (376h19m31.490614742s) [upload=301063]: Done uploading segments...
Jul 9 01:29:56.719991001 (376h19m31.490632336s) [upload=301063]: Upload finished. err: <nil>
Duration: 26.508080137s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.........................................X......XX.XX..X....] duration:
41.365949253s
FAIL (2025-07-09 01:30:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 41.365976937s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [................................X..........................X] duration:
1m0.003053261s
FAIL (2025-07-09 01:30:00 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
RequestItem_SegmentBegin
Jul 9 01:29:01.337433896 (376h18m36.108075261s) [seg_pos={0 0}, upload=301063]: Begin upload segment...
Jul 9 01:29:01.337487208 (376h18m36.108128543s) [seg_pos={0 0}, upload=301063]: Done begin upload segment.
Jul 9 01:29:01.337511635 (376h18m36.10815298s) [noise=true , address=77.90.31.11:4563 , piece_id=BC47C6JJI2QGZT54RFSVTB34IWFZVY2MYP4R55HYM3MC6PZ63L3Q, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.337520201 (376h18m36.108161546s) [noise=true , address=104.36.82.35:4614 , piece_id=JTXUCFS36ZNLCZHAEZORW77F3ZNAN5M6FYKQ2ZFYUAFXY5DRC3GQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.338221482 (376h18m36.108862827s) [noise=true , address=77.90.31.21:4909 , piece_id=BDRGEIFZ56SQ46453YLMKGZZE5F4FHIUJE35RU5RCUNDNQFXYSPQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.338235780 (376h18m36.108877145s) [noise=true , address=77.90.31.10:4533 , piece_id=CMUMWPDGLP2XWKFHCSXNE3IB7HYJEDOVYQNUM7DKI7THVBIHNDHQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.338701732 (376h18m36.109343067s) [noise=true , address=77.90.31.15:4701 , piece_id=M6US4USD57GPEL35W4IJF73X3BJMDRBLD2UXSHTLTYHMVI6XCTTA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.338733382 (376h18m36.109374717s) [noise=true , address=77.90.31.14:4682 , piece_id=KWYECXGPZ4TVAXVUK5IJHYRW36OFZGXFRWDLUSPJNZUUO5JDGLZA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339087590 (376h18m36.109728925s) [noise=true , address=77.90.31.17:4786 , piece_id=X4VZDLZC2AAXL5VRABJE5PIU6RJGJOQH67VHMSLT4VPLMF64QQ2A, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339094813 (376h18m36.109736159s) [noise=true , address=77.90.31.13:4651 , piece_id=TF3NGJ4AASQSXY4JSF27FCW7VSHF5VWYLHMNXHEQAMT5KJWXVNQQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339465213 (376h18m36.110106558s) [noise=true , address=77.90.31.12:4606 , piece_id=JCQCFDBA4YSIFSKS4RFYFUUYUAFU3XR2KI7VV4FCGFD4POFGXCHA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339475853 (376h18m36.110117198s) [noise=true , address=104.36.82.18:4391 , piece_id=FI7VKWZELHQFI2G4R5GOPAM4NLRT4VKGBJ26POQ2IS24F5EW77ZA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339842684 (376h18m36.110484019s) [noise=true , address=104.36.82.5:4868 , piece_id=W7Q3QS3IKB7X6TOOSKKMX5W42BHG274PXYNZ7UZCGSPNQXUFAURQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.339849308 (376h18m36.110490643s) [noise=true , address=77.90.31.10:4537 , piece_id=WQKTKYRY4C32EGPMHSXQAS3KSRAWIDDF7BYBZ2MKZOWHKFEXBGLA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.340181744 (376h18m36.110823079s) [noise=true , address=104.36.82.36:4651 , piece_id=QIMZ4CWZJCOYIIWEZGQYWZ2EHHL7DYAHHQ2YX5W7PPCH4QJNWS4A, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.340191722 (376h18m36.110833057s) [noise=true , address=104.36.82.19:4433 , piece_id=VUXOICADHIDKYIDNKJXDN3WA6FNDG3XJR3OCXC4UC26EA5P3NIMA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.340777823 (376h18m36.111419158s) [noise=true , address=77.90.31.11:4581 , piece_id=TKMT65PF3VOTCHFY6GBFC2W5JYOJTAHK5WFRZF3YAPOYHA7SGFDQ, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.341080963 (376h18m36.111722308s) [noise=true , address=104.36.82.44:4880 , piece_id=44M5JZ2MTKYBK6TH5HX46KAL24UCXIO4X4PV6DY6VUE4JHGKJXRA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.342101286 (376h18m36.112742631s) [noise=true , address=104.36.82.21:4477 , piece_id=N4BEJUNAM2RKLEZ7A7XKJI525EBWQ35UBMMSBZLIIC5AXNLMXCKA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:01.344027331 (376h18m36.114668667s) [noise=true , address=77.90.31.16:4757 , piece_id=HUJGTDFFEA5ATNB2HXCBERZYDL5N5BLMZYW6G3DASQTC2PM4IFYA, seg_pos={0 0}, upload=301063]: Uploading piece...
Jul 9 01:29:02.921702307 (376h18m37.692343642s) [upload=301063]: Next returned nil segment
Jul 9 01:29:02.921717205 (376h18m37.69235854s) [upload=301063]: Waiting for error group managing segments...
Jul 9 01:29:03.330770575 (376h18m38.10141191s) [noise=true , address=77.90.31.11:4581 , piece_id=TKMT65PF3VOTCHFY6GBFC2W5JYOJTAHK5WFRZF3YAPOYHA7SGFDQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.333021060 (376h18m38.103662395s) [noise=true , address=77.90.31.15:4701 , piece_id=M6US4USD57GPEL35W4IJF73X3BJMDRBLD2UXSHTLTYHMVI6XCTTA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.346762402 (376h18m38.117403737s) [noise=true , address=77.90.31.14:4682 , piece_id=KWYECXGPZ4TVAXVUK5IJHYRW36OFZGXFRWDLUSPJNZUUO5JDGLZA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.346815323 (376h18m38.117456658s) [noise=true , address=77.90.31.11:4563 , piece_id=BC47C6JJI2QGZT54RFSVTB34IWFZVY2MYP4R55HYM3MC6PZ63L3Q, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.347060412 (376h18m38.117701747s) [noise=true , address=77.90.31.17:4786 , piece_id=X4VZDLZC2AAXL5VRABJE5PIU6RJGJOQH67VHMSLT4VPLMF64QQ2A, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.347185391 (376h18m38.117826726s) [noise=true , address=77.90.31.21:4909 , piece_id=BDRGEIFZ56SQ46453YLMKGZZE5F4FHIUJE35RU5RCUNDNQFXYSPQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.347425851 (376h18m38.118067186s) [noise=true , address=77.90.31.13:4651 , piece_id=TF3NGJ4AASQSXY4JSF27FCW7VSHF5VWYLHMNXHEQAMT5KJWXVNQQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.347529449 (376h18m38.118170784s) [noise=true , address=77.90.31.12:4606 , piece_id=JCQCFDBA4YSIFSKS4RFYFUUYUAFU3XR2KI7VV4FCGFD4POFGXCHA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.347923393 (376h18m38.118564728s) [noise=true , address=77.90.31.10:4537 , piece_id=WQKTKYRY4C32EGPMHSXQAS3KSRAWIDDF7BYBZ2MKZOWHKFEXBGLA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.348515206 (376h18m38.119156541s) [noise=true , address=77.90.31.16:4757 , piece_id=HUJGTDFFEA5ATNB2HXCBERZYDL5N5BLMZYW6G3DASQTC2PM4IFYA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:03.363502611 (376h18m38.134143946s) [noise=true , address=77.90.31.10:4533 , piece_id=CMUMWPDGLP2XWKFHCSXNE3IB7HYJEDOVYQNUM7DKI7THVBIHNDHQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:26.444658884 (376h19m1.215300209s) [noise=true , address=104.36.82.19:4433 , piece_id=VUXOICADHIDKYIDNKJXDN3WA6FNDG3XJR3OCXC4UC26EA5P3NIMA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:37.476126743 (376h19m12.246768078s) [noise=true , address=104.36.82.36:4651 , piece_id=QIMZ4CWZJCOYIIWEZGQYWZ2EHHL7DYAHHQ2YX5W7PPCH4QJNWS4A, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:56.449247029 (376h19m31.219888364s) [noise=true , address=104.36.82.5:4868 , piece_id=W7Q3QS3IKB7X6TOOSKKMX5W42BHG274PXYNZ7UZCGSPNQXUFAURQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil>
Jul 9 01:29:56.449272748 (376h19m31.219914073s) [seg_pos={0 0}, upload=301063]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:29:56.449354975 (376h19m31.2199963s) [noise=true , address=104.36.82.21:4477 , piece_id=N4BEJUNAM2RKLEZ7A7XKJI525EBWQ35UBMMSBZLIIC5AXNLMXCKA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF
Jul 9 01:29:56.449457422 (376h19m31.220098757s) [noise=true , address=104.36.82.44:4880 , piece_id=44M5JZ2MTKYBK6TH5HX46KAL24UCXIO4X4PV6DY6VUE4JHGKJXRA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Jul 9 01:29:56.449486968 (376h19m31.220128303s) [noise=true , address=104.36.82.18:4391 , piece_id=FI7VKWZELHQFI2G4R5GOPAM4NLRT4VKGBJ26POQ2IS24F5EW77ZA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MHmmjheT4VDX93MiQNzQyTWkxZFzz2PwNvdRfsevZRbj6qmJS): protocol: write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection; write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection
Jul 9 01:29:56.449513970 (376h19m31.220155295s) [noise=true , address=104.36.82.35:4614 , piece_id=JTXUCFS36ZNLCZHAEZORW77F3ZNAN5M6FYKQ2ZFYUAFXY5DRC3GQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Pua5x8rAsuqYWCgoAY6fgy2SXNoDjzt7WwgMhzv1HN2KG5NY4): protocol: write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection; write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection
Jul 9 01:29:56.449528637 (376h19m31.220169972s) [seg_pos={0 0}, upload=301063]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:29:56.449590266 (376h19m31.220231601s) [upload=301063]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:29:56.449593853 (376h19m31.220235178s) [upload=301063]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:29:56.719973407 (376h19m31.490614742s) [upload=301063]: Done uploading segments...
Jul 9 01:29:56.719991001 (376h19m31.490632336s) [upload=301063]: Upload finished. err: <nil>
Duration: 1m0.003074682s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................X........................X.........] duration:
7.492267158s
FAIL (2025-07-09 01:29:52 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ommit
Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments...
Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil>
Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload
Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload
Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments...
Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false
Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment...
Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment.
Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment
Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments...
Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection
Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection
Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection
Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments...
Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 7.492296154s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.XX....X.....X...X......XX....X...X.X....X...X.X..X...X..X..] duration:
11.18461814s
FAIL (2025-07-09 01:29:47 UTC)
orbiter warning: context deadline exceeded
Duration: 11.184640304s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [..........................................X......XX.XX..X...] duration:
46.129441913s
FAIL (2025-07-09 01:29:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 46.129453114s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [..XX....X.....X...X......XX....X...X.X....X...X.X..X...X..X.] duration:
21.555817518s
FAIL (2025-07-09 01:29:02 UTC)
orbiter warning: context deadline exceeded
Duration: 21.555840441s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
26.496470447s
FAIL (2025-07-09 01:28:57 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
7:55.442899027 (375h57m30.213540362s) [upload=300780]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:07:55.885104488 (375h57m30.655745823s) [seg_pos={0 0}, upload=300780]: Begin upload segment...
Jul 9 01:07:55.885150927 (375h57m30.655792262s) [seg_pos={0 0}, upload=300780]: Done begin upload segment.
Jul 9 01:07:55.885170996 (375h57m30.655812331s) [noise=true , address=77.90.31.14:4669 , piece_id=KQNMOTC4GWOGYIZY2QK27LKEPN225LAYFJZWCJC6MRCK7QBPRFCQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885528510 (375h57m30.656169845s) [noise=true , address=104.36.82.23:4466 , piece_id=TTUJEA6JYTHG3WPYE5VLFPS6IYRXBUI5IZV3FV5725A6XPMJVZLQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885790180 (375h57m30.656431515s) [noise=true , address=77.90.31.13:4649 , piece_id=HXJ735B5UBJP4VSLABNV2RDSI6XELDRJ7DPXJXOZHU7QPGY6ON3Q, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885880773 (375h57m30.656522108s) [noise=true , address=77.90.31.16:4755 , piece_id=XAPAI7EP2B7XCJI6OBUVPPTPVAHARZVNVGUP4WBMLFPQNBOXUMBQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886081398 (375h57m30.656722733s) [noise=true , address=77.90.31.17:4801 , piece_id=K2UIXAWUISIA5KCKQCZMBJ3BHB2MS44MVMB7T4T2CCA2PMGJJDDA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886215583 (375h57m30.656856919s) [noise=true , address=104.36.82.39:4726 , piece_id=5OLQ7AGZ5I7AOKBCSP34VPYIFYB2IVWZJ6WWA4DTDJ5LRNXICUSQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886404385 (375h57m30.65704572s) [noise=true , address=45.91.116.9:4274 , piece_id=XKNACYYGKETXPQTRTATURZ2RRXLIDNUJOG2EKLG46OQUSMN3K33Q, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886513444 (375h57m30.657154769s) [noise=true , address=77.90.31.19:4861 , piece_id=RVU6MOFXHFWCAIKRNXKGQT4KADZSLIXXNIG6NDHHNJZ7KEBAXICA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886808428 (375h57m30.657449763s) [noise=true , address=104.36.82.3:4786 , piece_id=24V2D5IBAXBHUDVZ7KJXDNZHVHC2H6IWQ3NAISWJ5PZSPF4TOVWA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.887107310 (375h57m30.657748645s) [noise=true , address=104.36.82.17:4350 , piece_id=YDPD4LOKKFCGXAGBJGNL7D77APW27OHIYSWYUR7T6XX4ZFP3FHZA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.887393648 (375h57m30.658034983s) [noise=true , address=156.146.43.225:4439 , piece_id=WHIPMS22HXTUQAOUPNXT2DLA6XL2YWXM43KLEC55YFEY2SKBNIBA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888027160 (375h57m30.658668495s) [noise=true , address=104.36.82.18:4390 , piece_id=G24NDRJ3NCKCWG6OFNUVRXE6HRUDZ7I4P6AKQ446XBKHNIOCD2TQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888272730 (375h57m30.658914065s) [noise=true , address=77.90.31.12:4622 , piece_id=F566GJ7G6WUKJE63V7Z25TOJ25LDGSVYHPBYMQCQQLRDTRYBJFYQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888495466 (375h57m30.659136801s) [noise=true , address=77.90.31.13:4656 , piece_id=BM62XMC4FBDFBSF7BRQAUE2UWJZU5JUODULKP7CWVPRXFFCGS2CQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888749441 (375h57m30.659390776s) [noise=true , address=104.36.82.19:4411 , piece_id=CURSEVNLHJHEMVYVLGYHIVTHMYSZSEFLCR5ZDMFSQDREHNBLKSUA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889008307 (375h57m30.659649642s) [noise=true , address=77.90.31.11:4557 , piece_id=IPHAW23VXAVZ4XP3TOMX2RMVO2JDC65HLRRNBSMZTA5JJXY5EABQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889236964 (375h57m30.659878299s) [noise=true , address=77.90.31.18:4840 , piece_id=SNZW25XL4VJ7UN5ZMGU4ZSOW6FQA5ZEUPQCV5OKRSPQIA3F4XGSA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889458348 (375h57m30.660099683s) [noise=true , address=77.90.31.14:4677 , piece_id=53ZLF5H2KYVI6HM3GSHVC6JKGJ7TV3624DFNCXCK76ZIM5YVKLSQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:56.955856205 (375h57m31.72649754s) [noise=true , address=156.146.43.225:4439 , piece_id=WHIPMS22HXTUQAOUPNXT2DLA6XL2YWXM43KLEC55YFEY2SKBNIBA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12VztoHQmCpGLhyN8wKXqJ9u7BzG9UbbeQinmquUZ9Rb2e2w9Ze, address:156.146.43.225:4439): protocol: storage node overloaded, request limit: 2000
Jul 9 01:07:57.480276305 (375h57m32.25091764s) [upload=300780]: Next returned nil segment
Jul 9 01:07:57.480291304 (375h57m32.250932639s) [upload=300780]: Waiting for error group managing segments...
Jul 9 01:07:57.892917917 (375h57m32.663559252s) [noise=true , address=77.90.31.13:4649 , piece_id=HXJ735B5UBJP4VSLABNV2RDSI6XELDRJ7DPXJXOZHU7QPGY6ON3Q, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.894965904 (375h57m32.66560724s) [noise=true , address=77.90.31.19:4861 , piece_id=RVU6MOFXHFWCAIKRNXKGQT4KADZSLIXXNIG6NDHHNJZ7KEBAXICA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.895386990 (375h57m32.666028325s) [noise=true , address=77.90.31.13:4656 , piece_id=BM62XMC4FBDFBSF7BRQAUE2UWJZU5JUODULKP7CWVPRXFFCGS2CQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.897322493 (375h57m32.667963828s) [noise=true , address=77.90.31.11:4557 , piece_id=IPHAW23VXAVZ4XP3TOMX2RMVO2JDC65HLRRNBSMZTA5JJXY5EABQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.899196428 (375h57m32.669837753s) [noise=true , address=77.90.31.18:4840 , piece_id=SNZW25XL4VJ7UN5ZMGU4ZSOW6FQA5ZEUPQCV5OKRSPQIA3F4XGSA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.901992567 (375h57m32.672633902s) [noise=true , address=77.90.31.14:4677 , piece_id=53ZLF5H2KYVI6HM3GSHVC6JKGJ7TV3624DFNCXCK76ZIM5YVKLSQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.907178669 (375h57m32.677820014s) [noise=true , address=77.90.31.12:4622 , piece_id=F566GJ7G6WUKJE63V7Z25TOJ25LDGSVYHPBYMQCQQLRDTRYBJFYQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.917659831 (375h57m32.688301166s) [noise=true , address=77.90.31.14:4669 , piece_id=KQNMOTC4GWOGYIZY2QK27LKEPN225LAYFJZWCJC6MRCK7QBPRFCQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.928100055 (375h57m32.69874139s) [noise=true , address=77.90.31.17:4801 , piece_id=K2UIXAWUISIA5KCKQCZMBJ3BHB2MS44MVMB7T4T2CCA2PMGJJDDA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.955811390 (375h57m32.726452725s) [noise=true , address=77.90.31.16:4755 , piece_id=XAPAI7EP2B7XCJI6OBUVPPTPVAHARZVNVGUP4WBMLFPQNBOXUMBQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:08:00.166279605 (375h57m34.93692094s) [noise=true , address=45.91.116.9:4274 , piece_id=XKNACYYGKETXPQTRTATURZ2RRXLIDNUJOG2EKLG46OQUSMN3K33Q, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: storage node overloaded, request limit: 2000
Jul 9 01:08:10.737498024 (375h57m45.508139359s) [noise=true , address=104.36.82.39:4726 , piece_id=5OLQ7AGZ5I7AOKBCSP34VPYIFYB2IVWZJ6WWA4DTDJ5LRNXICUSQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:08:54.804118607 (375h58m29.574759942s) [noise=true , address=104.36.82.18:4390 , piece_id=G24NDRJ3NCKCWG6OFNUVRXE6HRUDZ7I4P6AKQ446XBKHNIOCD2TQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12AZCJHCR7rF4r62Bg7NcBkpNoLb6r6rEN2a5hZ9KNxJTtjV8jp, address:104.36.82.18:4390): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804258494 (375h58m29.574899819s) [noise=true , address=104.36.82.23:4466 , piece_id=TTUJEA6JYTHG3WPYE5VLFPS6IYRXBUI5IZV3FV5725A6XPMJVZLQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12rkNnk9RUhKBJ3EsakqMRb3wYwrhRLhB4ZYbrdfKPDk8erRJpP, address:104.36.82.23:4466): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804300485 (375h58m29.57494184s) [noise=true , address=104.36.82.17:4350 , piece_id=YDPD4LOKKFCGXAGBJGNL7D77APW27OHIYSWYUR7T6XX4ZFP3FHZA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804343667 (375h58m29.574985002s) [noise=true , address=104.36.82.3:4786 , piece_id=24V2D5IBAXBHUDVZ7KJXDNZHVHC2H6IWQ3NAISWJ5PZSPF4TOVWA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:127aTG7tofhLA5799hT6gvZZSkT1vvWWFejx7DS4Zcz8mBHHsuE, address:104.36.82.3:4786): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804418601 (375h58m29.575059936s) [noise=true , address=104.36.82.19:4411 , piece_id=CURSEVNLHJHEMVYVLGYHIVTHMYSZSEFLCR5ZDMFSQDREHNBLKSUA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12nradx4SMYLkuJw3Ad6o9Sq1iNtQR6GDu8qg7tCzZzNcTjVfyd, address:104.36.82.19:4411): protocol: write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection; write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection
Jul 9 01:08:54.804449420 (375h58m29.575090755s) [seg_pos={0 0}, upload=300780]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 9 01:08:55.034441116 (375h58m29.805082451s) [upload=300780]: Done uploading segments...
Jul 9 01:08:55.034460744 (375h58m29.805102079s) [upload=300780]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 26.496485526s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................X........................X........] duration:
5.872561466s
FAIL (2025-07-09 01:28:42 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ommit
Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments...
Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil>
Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload
Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload
Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments...
Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false
Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment...
Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment.
Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment
Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments...
Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection
Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection
Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection
Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments...
Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.872587215s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
42.755316302s
FAIL (2025-07-09 01:28:30 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
7:55.442899027 (375h57m30.213540362s) [upload=300780]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:07:55.885104488 (375h57m30.655745823s) [seg_pos={0 0}, upload=300780]: Begin upload segment...
Jul 9 01:07:55.885150927 (375h57m30.655792262s) [seg_pos={0 0}, upload=300780]: Done begin upload segment.
Jul 9 01:07:55.885170996 (375h57m30.655812331s) [noise=true , address=77.90.31.14:4669 , piece_id=KQNMOTC4GWOGYIZY2QK27LKEPN225LAYFJZWCJC6MRCK7QBPRFCQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885528510 (375h57m30.656169845s) [noise=true , address=104.36.82.23:4466 , piece_id=TTUJEA6JYTHG3WPYE5VLFPS6IYRXBUI5IZV3FV5725A6XPMJVZLQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885790180 (375h57m30.656431515s) [noise=true , address=77.90.31.13:4649 , piece_id=HXJ735B5UBJP4VSLABNV2RDSI6XELDRJ7DPXJXOZHU7QPGY6ON3Q, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885880773 (375h57m30.656522108s) [noise=true , address=77.90.31.16:4755 , piece_id=XAPAI7EP2B7XCJI6OBUVPPTPVAHARZVNVGUP4WBMLFPQNBOXUMBQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886081398 (375h57m30.656722733s) [noise=true , address=77.90.31.17:4801 , piece_id=K2UIXAWUISIA5KCKQCZMBJ3BHB2MS44MVMB7T4T2CCA2PMGJJDDA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886215583 (375h57m30.656856919s) [noise=true , address=104.36.82.39:4726 , piece_id=5OLQ7AGZ5I7AOKBCSP34VPYIFYB2IVWZJ6WWA4DTDJ5LRNXICUSQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886404385 (375h57m30.65704572s) [noise=true , address=45.91.116.9:4274 , piece_id=XKNACYYGKETXPQTRTATURZ2RRXLIDNUJOG2EKLG46OQUSMN3K33Q, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886513444 (375h57m30.657154769s) [noise=true , address=77.90.31.19:4861 , piece_id=RVU6MOFXHFWCAIKRNXKGQT4KADZSLIXXNIG6NDHHNJZ7KEBAXICA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886808428 (375h57m30.657449763s) [noise=true , address=104.36.82.3:4786 , piece_id=24V2D5IBAXBHUDVZ7KJXDNZHVHC2H6IWQ3NAISWJ5PZSPF4TOVWA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.887107310 (375h57m30.657748645s) [noise=true , address=104.36.82.17:4350 , piece_id=YDPD4LOKKFCGXAGBJGNL7D77APW27OHIYSWYUR7T6XX4ZFP3FHZA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.887393648 (375h57m30.658034983s) [noise=true , address=156.146.43.225:4439 , piece_id=WHIPMS22HXTUQAOUPNXT2DLA6XL2YWXM43KLEC55YFEY2SKBNIBA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888027160 (375h57m30.658668495s) [noise=true , address=104.36.82.18:4390 , piece_id=G24NDRJ3NCKCWG6OFNUVRXE6HRUDZ7I4P6AKQ446XBKHNIOCD2TQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888272730 (375h57m30.658914065s) [noise=true , address=77.90.31.12:4622 , piece_id=F566GJ7G6WUKJE63V7Z25TOJ25LDGSVYHPBYMQCQQLRDTRYBJFYQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888495466 (375h57m30.659136801s) [noise=true , address=77.90.31.13:4656 , piece_id=BM62XMC4FBDFBSF7BRQAUE2UWJZU5JUODULKP7CWVPRXFFCGS2CQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888749441 (375h57m30.659390776s) [noise=true , address=104.36.82.19:4411 , piece_id=CURSEVNLHJHEMVYVLGYHIVTHMYSZSEFLCR5ZDMFSQDREHNBLKSUA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889008307 (375h57m30.659649642s) [noise=true , address=77.90.31.11:4557 , piece_id=IPHAW23VXAVZ4XP3TOMX2RMVO2JDC65HLRRNBSMZTA5JJXY5EABQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889236964 (375h57m30.659878299s) [noise=true , address=77.90.31.18:4840 , piece_id=SNZW25XL4VJ7UN5ZMGU4ZSOW6FQA5ZEUPQCV5OKRSPQIA3F4XGSA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889458348 (375h57m30.660099683s) [noise=true , address=77.90.31.14:4677 , piece_id=53ZLF5H2KYVI6HM3GSHVC6JKGJ7TV3624DFNCXCK76ZIM5YVKLSQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:56.955856205 (375h57m31.72649754s) [noise=true , address=156.146.43.225:4439 , piece_id=WHIPMS22HXTUQAOUPNXT2DLA6XL2YWXM43KLEC55YFEY2SKBNIBA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12VztoHQmCpGLhyN8wKXqJ9u7BzG9UbbeQinmquUZ9Rb2e2w9Ze, address:156.146.43.225:4439): protocol: storage node overloaded, request limit: 2000
Jul 9 01:07:57.480276305 (375h57m32.25091764s) [upload=300780]: Next returned nil segment
Jul 9 01:07:57.480291304 (375h57m32.250932639s) [upload=300780]: Waiting for error group managing segments...
Jul 9 01:07:57.892917917 (375h57m32.663559252s) [noise=true , address=77.90.31.13:4649 , piece_id=HXJ735B5UBJP4VSLABNV2RDSI6XELDRJ7DPXJXOZHU7QPGY6ON3Q, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.894965904 (375h57m32.66560724s) [noise=true , address=77.90.31.19:4861 , piece_id=RVU6MOFXHFWCAIKRNXKGQT4KADZSLIXXNIG6NDHHNJZ7KEBAXICA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.895386990 (375h57m32.666028325s) [noise=true , address=77.90.31.13:4656 , piece_id=BM62XMC4FBDFBSF7BRQAUE2UWJZU5JUODULKP7CWVPRXFFCGS2CQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.897322493 (375h57m32.667963828s) [noise=true , address=77.90.31.11:4557 , piece_id=IPHAW23VXAVZ4XP3TOMX2RMVO2JDC65HLRRNBSMZTA5JJXY5EABQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.899196428 (375h57m32.669837753s) [noise=true , address=77.90.31.18:4840 , piece_id=SNZW25XL4VJ7UN5ZMGU4ZSOW6FQA5ZEUPQCV5OKRSPQIA3F4XGSA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.901992567 (375h57m32.672633902s) [noise=true , address=77.90.31.14:4677 , piece_id=53ZLF5H2KYVI6HM3GSHVC6JKGJ7TV3624DFNCXCK76ZIM5YVKLSQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.907178669 (375h57m32.677820014s) [noise=true , address=77.90.31.12:4622 , piece_id=F566GJ7G6WUKJE63V7Z25TOJ25LDGSVYHPBYMQCQQLRDTRYBJFYQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.917659831 (375h57m32.688301166s) [noise=true , address=77.90.31.14:4669 , piece_id=KQNMOTC4GWOGYIZY2QK27LKEPN225LAYFJZWCJC6MRCK7QBPRFCQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.928100055 (375h57m32.69874139s) [noise=true , address=77.90.31.17:4801 , piece_id=K2UIXAWUISIA5KCKQCZMBJ3BHB2MS44MVMB7T4T2CCA2PMGJJDDA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.955811390 (375h57m32.726452725s) [noise=true , address=77.90.31.16:4755 , piece_id=XAPAI7EP2B7XCJI6OBUVPPTPVAHARZVNVGUP4WBMLFPQNBOXUMBQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:08:00.166279605 (375h57m34.93692094s) [noise=true , address=45.91.116.9:4274 , piece_id=XKNACYYGKETXPQTRTATURZ2RRXLIDNUJOG2EKLG46OQUSMN3K33Q, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: storage node overloaded, request limit: 2000
Jul 9 01:08:10.737498024 (375h57m45.508139359s) [noise=true , address=104.36.82.39:4726 , piece_id=5OLQ7AGZ5I7AOKBCSP34VPYIFYB2IVWZJ6WWA4DTDJ5LRNXICUSQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:08:54.804118607 (375h58m29.574759942s) [noise=true , address=104.36.82.18:4390 , piece_id=G24NDRJ3NCKCWG6OFNUVRXE6HRUDZ7I4P6AKQ446XBKHNIOCD2TQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12AZCJHCR7rF4r62Bg7NcBkpNoLb6r6rEN2a5hZ9KNxJTtjV8jp, address:104.36.82.18:4390): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804258494 (375h58m29.574899819s) [noise=true , address=104.36.82.23:4466 , piece_id=TTUJEA6JYTHG3WPYE5VLFPS6IYRXBUI5IZV3FV5725A6XPMJVZLQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12rkNnk9RUhKBJ3EsakqMRb3wYwrhRLhB4ZYbrdfKPDk8erRJpP, address:104.36.82.23:4466): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804300485 (375h58m29.57494184s) [noise=true , address=104.36.82.17:4350 , piece_id=YDPD4LOKKFCGXAGBJGNL7D77APW27OHIYSWYUR7T6XX4ZFP3FHZA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804343667 (375h58m29.574985002s) [noise=true , address=104.36.82.3:4786 , piece_id=24V2D5IBAXBHUDVZ7KJXDNZHVHC2H6IWQ3NAISWJ5PZSPF4TOVWA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:127aTG7tofhLA5799hT6gvZZSkT1vvWWFejx7DS4Zcz8mBHHsuE, address:104.36.82.3:4786): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804418601 (375h58m29.575059936s) [noise=true , address=104.36.82.19:4411 , piece_id=CURSEVNLHJHEMVYVLGYHIVTHMYSZSEFLCR5ZDMFSQDREHNBLKSUA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12nradx4SMYLkuJw3Ad6o9Sq1iNtQR6GDu8qg7tCzZzNcTjVfyd, address:104.36.82.19:4411): protocol: write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection; write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection
Jul 9 01:08:54.804449420 (375h58m29.575090755s) [seg_pos={0 0}, upload=300780]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 9 01:08:55.034441116 (375h58m29.805082451s) [upload=300780]: Done uploading segments...
Jul 9 01:08:55.034460744 (375h58m29.805102079s) [upload=300780]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 42.755337522s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [...........................................X......XX.XX..X..] duration:
57.702083528s
FAIL (2025-07-09 01:28:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 57.702105189s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [...XX....X.....X...X......XX....X...X.X....X...X.X..X...X..X] duration:
30.000807969s
FAIL (2025-07-09 01:28:03 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000829081s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................X........................X.......] duration:
5.796208719s
FAIL (2025-07-09 01:27:53 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ommit
Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments...
Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil>
Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload
Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload
Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments...
Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false
Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment...
Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment.
Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment
Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments...
Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection
Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection
Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection
Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments...
Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.796238337s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
28.101880909s
FAIL (2025-07-09 01:27:41 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
7:55.442899027 (375h57m30.213540362s) [upload=300780]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:07:55.885104488 (375h57m30.655745823s) [seg_pos={0 0}, upload=300780]: Begin upload segment...
Jul 9 01:07:55.885150927 (375h57m30.655792262s) [seg_pos={0 0}, upload=300780]: Done begin upload segment.
Jul 9 01:07:55.885170996 (375h57m30.655812331s) [noise=true , address=77.90.31.14:4669 , piece_id=KQNMOTC4GWOGYIZY2QK27LKEPN225LAYFJZWCJC6MRCK7QBPRFCQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885528510 (375h57m30.656169845s) [noise=true , address=104.36.82.23:4466 , piece_id=TTUJEA6JYTHG3WPYE5VLFPS6IYRXBUI5IZV3FV5725A6XPMJVZLQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885790180 (375h57m30.656431515s) [noise=true , address=77.90.31.13:4649 , piece_id=HXJ735B5UBJP4VSLABNV2RDSI6XELDRJ7DPXJXOZHU7QPGY6ON3Q, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885880773 (375h57m30.656522108s) [noise=true , address=77.90.31.16:4755 , piece_id=XAPAI7EP2B7XCJI6OBUVPPTPVAHARZVNVGUP4WBMLFPQNBOXUMBQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886081398 (375h57m30.656722733s) [noise=true , address=77.90.31.17:4801 , piece_id=K2UIXAWUISIA5KCKQCZMBJ3BHB2MS44MVMB7T4T2CCA2PMGJJDDA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886215583 (375h57m30.656856919s) [noise=true , address=104.36.82.39:4726 , piece_id=5OLQ7AGZ5I7AOKBCSP34VPYIFYB2IVWZJ6WWA4DTDJ5LRNXICUSQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886404385 (375h57m30.65704572s) [noise=true , address=45.91.116.9:4274 , piece_id=XKNACYYGKETXPQTRTATURZ2RRXLIDNUJOG2EKLG46OQUSMN3K33Q, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886513444 (375h57m30.657154769s) [noise=true , address=77.90.31.19:4861 , piece_id=RVU6MOFXHFWCAIKRNXKGQT4KADZSLIXXNIG6NDHHNJZ7KEBAXICA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886808428 (375h57m30.657449763s) [noise=true , address=104.36.82.3:4786 , piece_id=24V2D5IBAXBHUDVZ7KJXDNZHVHC2H6IWQ3NAISWJ5PZSPF4TOVWA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.887107310 (375h57m30.657748645s) [noise=true , address=104.36.82.17:4350 , piece_id=YDPD4LOKKFCGXAGBJGNL7D77APW27OHIYSWYUR7T6XX4ZFP3FHZA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.887393648 (375h57m30.658034983s) [noise=true , address=156.146.43.225:4439 , piece_id=WHIPMS22HXTUQAOUPNXT2DLA6XL2YWXM43KLEC55YFEY2SKBNIBA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888027160 (375h57m30.658668495s) [noise=true , address=104.36.82.18:4390 , piece_id=G24NDRJ3NCKCWG6OFNUVRXE6HRUDZ7I4P6AKQ446XBKHNIOCD2TQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888272730 (375h57m30.658914065s) [noise=true , address=77.90.31.12:4622 , piece_id=F566GJ7G6WUKJE63V7Z25TOJ25LDGSVYHPBYMQCQQLRDTRYBJFYQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888495466 (375h57m30.659136801s) [noise=true , address=77.90.31.13:4656 , piece_id=BM62XMC4FBDFBSF7BRQAUE2UWJZU5JUODULKP7CWVPRXFFCGS2CQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888749441 (375h57m30.659390776s) [noise=true , address=104.36.82.19:4411 , piece_id=CURSEVNLHJHEMVYVLGYHIVTHMYSZSEFLCR5ZDMFSQDREHNBLKSUA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889008307 (375h57m30.659649642s) [noise=true , address=77.90.31.11:4557 , piece_id=IPHAW23VXAVZ4XP3TOMX2RMVO2JDC65HLRRNBSMZTA5JJXY5EABQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889236964 (375h57m30.659878299s) [noise=true , address=77.90.31.18:4840 , piece_id=SNZW25XL4VJ7UN5ZMGU4ZSOW6FQA5ZEUPQCV5OKRSPQIA3F4XGSA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889458348 (375h57m30.660099683s) [noise=true , address=77.90.31.14:4677 , piece_id=53ZLF5H2KYVI6HM3GSHVC6JKGJ7TV3624DFNCXCK76ZIM5YVKLSQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:56.955856205 (375h57m31.72649754s) [noise=true , address=156.146.43.225:4439 , piece_id=WHIPMS22HXTUQAOUPNXT2DLA6XL2YWXM43KLEC55YFEY2SKBNIBA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12VztoHQmCpGLhyN8wKXqJ9u7BzG9UbbeQinmquUZ9Rb2e2w9Ze, address:156.146.43.225:4439): protocol: storage node overloaded, request limit: 2000
Jul 9 01:07:57.480276305 (375h57m32.25091764s) [upload=300780]: Next returned nil segment
Jul 9 01:07:57.480291304 (375h57m32.250932639s) [upload=300780]: Waiting for error group managing segments...
Jul 9 01:07:57.892917917 (375h57m32.663559252s) [noise=true , address=77.90.31.13:4649 , piece_id=HXJ735B5UBJP4VSLABNV2RDSI6XELDRJ7DPXJXOZHU7QPGY6ON3Q, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.894965904 (375h57m32.66560724s) [noise=true , address=77.90.31.19:4861 , piece_id=RVU6MOFXHFWCAIKRNXKGQT4KADZSLIXXNIG6NDHHNJZ7KEBAXICA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.895386990 (375h57m32.666028325s) [noise=true , address=77.90.31.13:4656 , piece_id=BM62XMC4FBDFBSF7BRQAUE2UWJZU5JUODULKP7CWVPRXFFCGS2CQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.897322493 (375h57m32.667963828s) [noise=true , address=77.90.31.11:4557 , piece_id=IPHAW23VXAVZ4XP3TOMX2RMVO2JDC65HLRRNBSMZTA5JJXY5EABQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.899196428 (375h57m32.669837753s) [noise=true , address=77.90.31.18:4840 , piece_id=SNZW25XL4VJ7UN5ZMGU4ZSOW6FQA5ZEUPQCV5OKRSPQIA3F4XGSA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.901992567 (375h57m32.672633902s) [noise=true , address=77.90.31.14:4677 , piece_id=53ZLF5H2KYVI6HM3GSHVC6JKGJ7TV3624DFNCXCK76ZIM5YVKLSQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.907178669 (375h57m32.677820014s) [noise=true , address=77.90.31.12:4622 , piece_id=F566GJ7G6WUKJE63V7Z25TOJ25LDGSVYHPBYMQCQQLRDTRYBJFYQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.917659831 (375h57m32.688301166s) [noise=true , address=77.90.31.14:4669 , piece_id=KQNMOTC4GWOGYIZY2QK27LKEPN225LAYFJZWCJC6MRCK7QBPRFCQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.928100055 (375h57m32.69874139s) [noise=true , address=77.90.31.17:4801 , piece_id=K2UIXAWUISIA5KCKQCZMBJ3BHB2MS44MVMB7T4T2CCA2PMGJJDDA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.955811390 (375h57m32.726452725s) [noise=true , address=77.90.31.16:4755 , piece_id=XAPAI7EP2B7XCJI6OBUVPPTPVAHARZVNVGUP4WBMLFPQNBOXUMBQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:08:00.166279605 (375h57m34.93692094s) [noise=true , address=45.91.116.9:4274 , piece_id=XKNACYYGKETXPQTRTATURZ2RRXLIDNUJOG2EKLG46OQUSMN3K33Q, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: storage node overloaded, request limit: 2000
Jul 9 01:08:10.737498024 (375h57m45.508139359s) [noise=true , address=104.36.82.39:4726 , piece_id=5OLQ7AGZ5I7AOKBCSP34VPYIFYB2IVWZJ6WWA4DTDJ5LRNXICUSQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:08:54.804118607 (375h58m29.574759942s) [noise=true , address=104.36.82.18:4390 , piece_id=G24NDRJ3NCKCWG6OFNUVRXE6HRUDZ7I4P6AKQ446XBKHNIOCD2TQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12AZCJHCR7rF4r62Bg7NcBkpNoLb6r6rEN2a5hZ9KNxJTtjV8jp, address:104.36.82.18:4390): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804258494 (375h58m29.574899819s) [noise=true , address=104.36.82.23:4466 , piece_id=TTUJEA6JYTHG3WPYE5VLFPS6IYRXBUI5IZV3FV5725A6XPMJVZLQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12rkNnk9RUhKBJ3EsakqMRb3wYwrhRLhB4ZYbrdfKPDk8erRJpP, address:104.36.82.23:4466): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804300485 (375h58m29.57494184s) [noise=true , address=104.36.82.17:4350 , piece_id=YDPD4LOKKFCGXAGBJGNL7D77APW27OHIYSWYUR7T6XX4ZFP3FHZA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804343667 (375h58m29.574985002s) [noise=true , address=104.36.82.3:4786 , piece_id=24V2D5IBAXBHUDVZ7KJXDNZHVHC2H6IWQ3NAISWJ5PZSPF4TOVWA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:127aTG7tofhLA5799hT6gvZZSkT1vvWWFejx7DS4Zcz8mBHHsuE, address:104.36.82.3:4786): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804418601 (375h58m29.575059936s) [noise=true , address=104.36.82.19:4411 , piece_id=CURSEVNLHJHEMVYVLGYHIVTHMYSZSEFLCR5ZDMFSQDREHNBLKSUA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12nradx4SMYLkuJw3Ad6o9Sq1iNtQR6GDu8qg7tCzZzNcTjVfyd, address:104.36.82.19:4411): protocol: write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection; write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection
Jul 9 01:08:54.804449420 (375h58m29.575090755s) [seg_pos={0 0}, upload=300780]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 9 01:08:55.034441116 (375h58m29.805082451s) [upload=300780]: Done uploading segments...
Jul 9 01:08:55.034460744 (375h58m29.805102079s) [upload=300780]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 28.101908861s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [X...XX....X.....X...X......XX....X...X.X....X...X.X..X...X..] duration:
7.388526723s
FAIL (2025-07-09 01:27:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.38855121s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................X........................X......] duration:
5.853447555s
FAIL (2025-07-09 01:27:20 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ommit
Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments...
Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil>
Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload
Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload
Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments...
Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false
Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment...
Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment.
Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment
Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments...
Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection
Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection
Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection
Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments...
Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.853485989s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [............................................X......XX.XX..X.] duration:
51.419619096s
FAIL (2025-07-09 01:27:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 51.419632372s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
36.866722116s
FAIL (2025-07-09 01:26:43 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
7:55.442899027 (375h57m30.213540362s) [upload=300780]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:07:55.885104488 (375h57m30.655745823s) [seg_pos={0 0}, upload=300780]: Begin upload segment...
Jul 9 01:07:55.885150927 (375h57m30.655792262s) [seg_pos={0 0}, upload=300780]: Done begin upload segment.
Jul 9 01:07:55.885170996 (375h57m30.655812331s) [noise=true , address=77.90.31.14:4669 , piece_id=KQNMOTC4GWOGYIZY2QK27LKEPN225LAYFJZWCJC6MRCK7QBPRFCQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885528510 (375h57m30.656169845s) [noise=true , address=104.36.82.23:4466 , piece_id=TTUJEA6JYTHG3WPYE5VLFPS6IYRXBUI5IZV3FV5725A6XPMJVZLQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885790180 (375h57m30.656431515s) [noise=true , address=77.90.31.13:4649 , piece_id=HXJ735B5UBJP4VSLABNV2RDSI6XELDRJ7DPXJXOZHU7QPGY6ON3Q, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885880773 (375h57m30.656522108s) [noise=true , address=77.90.31.16:4755 , piece_id=XAPAI7EP2B7XCJI6OBUVPPTPVAHARZVNVGUP4WBMLFPQNBOXUMBQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886081398 (375h57m30.656722733s) [noise=true , address=77.90.31.17:4801 , piece_id=K2UIXAWUISIA5KCKQCZMBJ3BHB2MS44MVMB7T4T2CCA2PMGJJDDA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886215583 (375h57m30.656856919s) [noise=true , address=104.36.82.39:4726 , piece_id=5OLQ7AGZ5I7AOKBCSP34VPYIFYB2IVWZJ6WWA4DTDJ5LRNXICUSQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886404385 (375h57m30.65704572s) [noise=true , address=45.91.116.9:4274 , piece_id=XKNACYYGKETXPQTRTATURZ2RRXLIDNUJOG2EKLG46OQUSMN3K33Q, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886513444 (375h57m30.657154769s) [noise=true , address=77.90.31.19:4861 , piece_id=RVU6MOFXHFWCAIKRNXKGQT4KADZSLIXXNIG6NDHHNJZ7KEBAXICA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886808428 (375h57m30.657449763s) [noise=true , address=104.36.82.3:4786 , piece_id=24V2D5IBAXBHUDVZ7KJXDNZHVHC2H6IWQ3NAISWJ5PZSPF4TOVWA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.887107310 (375h57m30.657748645s) [noise=true , address=104.36.82.17:4350 , piece_id=YDPD4LOKKFCGXAGBJGNL7D77APW27OHIYSWYUR7T6XX4ZFP3FHZA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.887393648 (375h57m30.658034983s) [noise=true , address=156.146.43.225:4439 , piece_id=WHIPMS22HXTUQAOUPNXT2DLA6XL2YWXM43KLEC55YFEY2SKBNIBA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888027160 (375h57m30.658668495s) [noise=true , address=104.36.82.18:4390 , piece_id=G24NDRJ3NCKCWG6OFNUVRXE6HRUDZ7I4P6AKQ446XBKHNIOCD2TQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888272730 (375h57m30.658914065s) [noise=true , address=77.90.31.12:4622 , piece_id=F566GJ7G6WUKJE63V7Z25TOJ25LDGSVYHPBYMQCQQLRDTRYBJFYQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888495466 (375h57m30.659136801s) [noise=true , address=77.90.31.13:4656 , piece_id=BM62XMC4FBDFBSF7BRQAUE2UWJZU5JUODULKP7CWVPRXFFCGS2CQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888749441 (375h57m30.659390776s) [noise=true , address=104.36.82.19:4411 , piece_id=CURSEVNLHJHEMVYVLGYHIVTHMYSZSEFLCR5ZDMFSQDREHNBLKSUA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889008307 (375h57m30.659649642s) [noise=true , address=77.90.31.11:4557 , piece_id=IPHAW23VXAVZ4XP3TOMX2RMVO2JDC65HLRRNBSMZTA5JJXY5EABQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889236964 (375h57m30.659878299s) [noise=true , address=77.90.31.18:4840 , piece_id=SNZW25XL4VJ7UN5ZMGU4ZSOW6FQA5ZEUPQCV5OKRSPQIA3F4XGSA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889458348 (375h57m30.660099683s) [noise=true , address=77.90.31.14:4677 , piece_id=53ZLF5H2KYVI6HM3GSHVC6JKGJ7TV3624DFNCXCK76ZIM5YVKLSQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:56.955856205 (375h57m31.72649754s) [noise=true , address=156.146.43.225:4439 , piece_id=WHIPMS22HXTUQAOUPNXT2DLA6XL2YWXM43KLEC55YFEY2SKBNIBA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12VztoHQmCpGLhyN8wKXqJ9u7BzG9UbbeQinmquUZ9Rb2e2w9Ze, address:156.146.43.225:4439): protocol: storage node overloaded, request limit: 2000
Jul 9 01:07:57.480276305 (375h57m32.25091764s) [upload=300780]: Next returned nil segment
Jul 9 01:07:57.480291304 (375h57m32.250932639s) [upload=300780]: Waiting for error group managing segments...
Jul 9 01:07:57.892917917 (375h57m32.663559252s) [noise=true , address=77.90.31.13:4649 , piece_id=HXJ735B5UBJP4VSLABNV2RDSI6XELDRJ7DPXJXOZHU7QPGY6ON3Q, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.894965904 (375h57m32.66560724s) [noise=true , address=77.90.31.19:4861 , piece_id=RVU6MOFXHFWCAIKRNXKGQT4KADZSLIXXNIG6NDHHNJZ7KEBAXICA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.895386990 (375h57m32.666028325s) [noise=true , address=77.90.31.13:4656 , piece_id=BM62XMC4FBDFBSF7BRQAUE2UWJZU5JUODULKP7CWVPRXFFCGS2CQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.897322493 (375h57m32.667963828s) [noise=true , address=77.90.31.11:4557 , piece_id=IPHAW23VXAVZ4XP3TOMX2RMVO2JDC65HLRRNBSMZTA5JJXY5EABQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.899196428 (375h57m32.669837753s) [noise=true , address=77.90.31.18:4840 , piece_id=SNZW25XL4VJ7UN5ZMGU4ZSOW6FQA5ZEUPQCV5OKRSPQIA3F4XGSA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.901992567 (375h57m32.672633902s) [noise=true , address=77.90.31.14:4677 , piece_id=53ZLF5H2KYVI6HM3GSHVC6JKGJ7TV3624DFNCXCK76ZIM5YVKLSQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.907178669 (375h57m32.677820014s) [noise=true , address=77.90.31.12:4622 , piece_id=F566GJ7G6WUKJE63V7Z25TOJ25LDGSVYHPBYMQCQQLRDTRYBJFYQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.917659831 (375h57m32.688301166s) [noise=true , address=77.90.31.14:4669 , piece_id=KQNMOTC4GWOGYIZY2QK27LKEPN225LAYFJZWCJC6MRCK7QBPRFCQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.928100055 (375h57m32.69874139s) [noise=true , address=77.90.31.17:4801 , piece_id=K2UIXAWUISIA5KCKQCZMBJ3BHB2MS44MVMB7T4T2CCA2PMGJJDDA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.955811390 (375h57m32.726452725s) [noise=true , address=77.90.31.16:4755 , piece_id=XAPAI7EP2B7XCJI6OBUVPPTPVAHARZVNVGUP4WBMLFPQNBOXUMBQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:08:00.166279605 (375h57m34.93692094s) [noise=true , address=45.91.116.9:4274 , piece_id=XKNACYYGKETXPQTRTATURZ2RRXLIDNUJOG2EKLG46OQUSMN3K33Q, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: storage node overloaded, request limit: 2000
Jul 9 01:08:10.737498024 (375h57m45.508139359s) [noise=true , address=104.36.82.39:4726 , piece_id=5OLQ7AGZ5I7AOKBCSP34VPYIFYB2IVWZJ6WWA4DTDJ5LRNXICUSQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:08:54.804118607 (375h58m29.574759942s) [noise=true , address=104.36.82.18:4390 , piece_id=G24NDRJ3NCKCWG6OFNUVRXE6HRUDZ7I4P6AKQ446XBKHNIOCD2TQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12AZCJHCR7rF4r62Bg7NcBkpNoLb6r6rEN2a5hZ9KNxJTtjV8jp, address:104.36.82.18:4390): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804258494 (375h58m29.574899819s) [noise=true , address=104.36.82.23:4466 , piece_id=TTUJEA6JYTHG3WPYE5VLFPS6IYRXBUI5IZV3FV5725A6XPMJVZLQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12rkNnk9RUhKBJ3EsakqMRb3wYwrhRLhB4ZYbrdfKPDk8erRJpP, address:104.36.82.23:4466): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804300485 (375h58m29.57494184s) [noise=true , address=104.36.82.17:4350 , piece_id=YDPD4LOKKFCGXAGBJGNL7D77APW27OHIYSWYUR7T6XX4ZFP3FHZA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804343667 (375h58m29.574985002s) [noise=true , address=104.36.82.3:4786 , piece_id=24V2D5IBAXBHUDVZ7KJXDNZHVHC2H6IWQ3NAISWJ5PZSPF4TOVWA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:127aTG7tofhLA5799hT6gvZZSkT1vvWWFejx7DS4Zcz8mBHHsuE, address:104.36.82.3:4786): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804418601 (375h58m29.575059936s) [noise=true , address=104.36.82.19:4411 , piece_id=CURSEVNLHJHEMVYVLGYHIVTHMYSZSEFLCR5ZDMFSQDREHNBLKSUA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12nradx4SMYLkuJw3Ad6o9Sq1iNtQR6GDu8qg7tCzZzNcTjVfyd, address:104.36.82.19:4411): protocol: write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection; write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection
Jul 9 01:08:54.804449420 (375h58m29.575090755s) [seg_pos={0 0}, upload=300780]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 9 01:08:55.034441116 (375h58m29.805082451s) [upload=300780]: Done uploading segments...
Jul 9 01:08:55.034460744 (375h58m29.805102079s) [upload=300780]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 36.866741964s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.X...XX....X.....X...X......XX....X...X.X....X...X.X..X...X.] duration:
6.315718659s
FAIL (2025-07-09 01:26:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.315740401s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.............................................X......XX.XX..X] duration:
1m0.000503632s
FAIL (2025-07-09 01:26:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000524121s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................X........................X.....] duration:
5.837579937s
FAIL (2025-07-09 01:26:17 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ommit
Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments...
Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil>
Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload
Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload
Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments...
Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false
Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment...
Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment.
Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment
Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments...
Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection
Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection
Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection
Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments...
Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.837607249s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
25.249608279s
FAIL (2025-07-09 01:26:06 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
7:55.442899027 (375h57m30.213540362s) [upload=300780]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:07:55.885104488 (375h57m30.655745823s) [seg_pos={0 0}, upload=300780]: Begin upload segment...
Jul 9 01:07:55.885150927 (375h57m30.655792262s) [seg_pos={0 0}, upload=300780]: Done begin upload segment.
Jul 9 01:07:55.885170996 (375h57m30.655812331s) [noise=true , address=77.90.31.14:4669 , piece_id=KQNMOTC4GWOGYIZY2QK27LKEPN225LAYFJZWCJC6MRCK7QBPRFCQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885528510 (375h57m30.656169845s) [noise=true , address=104.36.82.23:4466 , piece_id=TTUJEA6JYTHG3WPYE5VLFPS6IYRXBUI5IZV3FV5725A6XPMJVZLQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885790180 (375h57m30.656431515s) [noise=true , address=77.90.31.13:4649 , piece_id=HXJ735B5UBJP4VSLABNV2RDSI6XELDRJ7DPXJXOZHU7QPGY6ON3Q, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885880773 (375h57m30.656522108s) [noise=true , address=77.90.31.16:4755 , piece_id=XAPAI7EP2B7XCJI6OBUVPPTPVAHARZVNVGUP4WBMLFPQNBOXUMBQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886081398 (375h57m30.656722733s) [noise=true , address=77.90.31.17:4801 , piece_id=K2UIXAWUISIA5KCKQCZMBJ3BHB2MS44MVMB7T4T2CCA2PMGJJDDA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886215583 (375h57m30.656856919s) [noise=true , address=104.36.82.39:4726 , piece_id=5OLQ7AGZ5I7AOKBCSP34VPYIFYB2IVWZJ6WWA4DTDJ5LRNXICUSQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886404385 (375h57m30.65704572s) [noise=true , address=45.91.116.9:4274 , piece_id=XKNACYYGKETXPQTRTATURZ2RRXLIDNUJOG2EKLG46OQUSMN3K33Q, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886513444 (375h57m30.657154769s) [noise=true , address=77.90.31.19:4861 , piece_id=RVU6MOFXHFWCAIKRNXKGQT4KADZSLIXXNIG6NDHHNJZ7KEBAXICA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886808428 (375h57m30.657449763s) [noise=true , address=104.36.82.3:4786 , piece_id=24V2D5IBAXBHUDVZ7KJXDNZHVHC2H6IWQ3NAISWJ5PZSPF4TOVWA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.887107310 (375h57m30.657748645s) [noise=true , address=104.36.82.17:4350 , piece_id=YDPD4LOKKFCGXAGBJGNL7D77APW27OHIYSWYUR7T6XX4ZFP3FHZA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.887393648 (375h57m30.658034983s) [noise=true , address=156.146.43.225:4439 , piece_id=WHIPMS22HXTUQAOUPNXT2DLA6XL2YWXM43KLEC55YFEY2SKBNIBA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888027160 (375h57m30.658668495s) [noise=true , address=104.36.82.18:4390 , piece_id=G24NDRJ3NCKCWG6OFNUVRXE6HRUDZ7I4P6AKQ446XBKHNIOCD2TQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888272730 (375h57m30.658914065s) [noise=true , address=77.90.31.12:4622 , piece_id=F566GJ7G6WUKJE63V7Z25TOJ25LDGSVYHPBYMQCQQLRDTRYBJFYQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888495466 (375h57m30.659136801s) [noise=true , address=77.90.31.13:4656 , piece_id=BM62XMC4FBDFBSF7BRQAUE2UWJZU5JUODULKP7CWVPRXFFCGS2CQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888749441 (375h57m30.659390776s) [noise=true , address=104.36.82.19:4411 , piece_id=CURSEVNLHJHEMVYVLGYHIVTHMYSZSEFLCR5ZDMFSQDREHNBLKSUA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889008307 (375h57m30.659649642s) [noise=true , address=77.90.31.11:4557 , piece_id=IPHAW23VXAVZ4XP3TOMX2RMVO2JDC65HLRRNBSMZTA5JJXY5EABQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889236964 (375h57m30.659878299s) [noise=true , address=77.90.31.18:4840 , piece_id=SNZW25XL4VJ7UN5ZMGU4ZSOW6FQA5ZEUPQCV5OKRSPQIA3F4XGSA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889458348 (375h57m30.660099683s) [noise=true , address=77.90.31.14:4677 , piece_id=53ZLF5H2KYVI6HM3GSHVC6JKGJ7TV3624DFNCXCK76ZIM5YVKLSQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:56.955856205 (375h57m31.72649754s) [noise=true , address=156.146.43.225:4439 , piece_id=WHIPMS22HXTUQAOUPNXT2DLA6XL2YWXM43KLEC55YFEY2SKBNIBA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12VztoHQmCpGLhyN8wKXqJ9u7BzG9UbbeQinmquUZ9Rb2e2w9Ze, address:156.146.43.225:4439): protocol: storage node overloaded, request limit: 2000
Jul 9 01:07:57.480276305 (375h57m32.25091764s) [upload=300780]: Next returned nil segment
Jul 9 01:07:57.480291304 (375h57m32.250932639s) [upload=300780]: Waiting for error group managing segments...
Jul 9 01:07:57.892917917 (375h57m32.663559252s) [noise=true , address=77.90.31.13:4649 , piece_id=HXJ735B5UBJP4VSLABNV2RDSI6XELDRJ7DPXJXOZHU7QPGY6ON3Q, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.894965904 (375h57m32.66560724s) [noise=true , address=77.90.31.19:4861 , piece_id=RVU6MOFXHFWCAIKRNXKGQT4KADZSLIXXNIG6NDHHNJZ7KEBAXICA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.895386990 (375h57m32.666028325s) [noise=true , address=77.90.31.13:4656 , piece_id=BM62XMC4FBDFBSF7BRQAUE2UWJZU5JUODULKP7CWVPRXFFCGS2CQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.897322493 (375h57m32.667963828s) [noise=true , address=77.90.31.11:4557 , piece_id=IPHAW23VXAVZ4XP3TOMX2RMVO2JDC65HLRRNBSMZTA5JJXY5EABQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.899196428 (375h57m32.669837753s) [noise=true , address=77.90.31.18:4840 , piece_id=SNZW25XL4VJ7UN5ZMGU4ZSOW6FQA5ZEUPQCV5OKRSPQIA3F4XGSA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.901992567 (375h57m32.672633902s) [noise=true , address=77.90.31.14:4677 , piece_id=53ZLF5H2KYVI6HM3GSHVC6JKGJ7TV3624DFNCXCK76ZIM5YVKLSQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.907178669 (375h57m32.677820014s) [noise=true , address=77.90.31.12:4622 , piece_id=F566GJ7G6WUKJE63V7Z25TOJ25LDGSVYHPBYMQCQQLRDTRYBJFYQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.917659831 (375h57m32.688301166s) [noise=true , address=77.90.31.14:4669 , piece_id=KQNMOTC4GWOGYIZY2QK27LKEPN225LAYFJZWCJC6MRCK7QBPRFCQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.928100055 (375h57m32.69874139s) [noise=true , address=77.90.31.17:4801 , piece_id=K2UIXAWUISIA5KCKQCZMBJ3BHB2MS44MVMB7T4T2CCA2PMGJJDDA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.955811390 (375h57m32.726452725s) [noise=true , address=77.90.31.16:4755 , piece_id=XAPAI7EP2B7XCJI6OBUVPPTPVAHARZVNVGUP4WBMLFPQNBOXUMBQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:08:00.166279605 (375h57m34.93692094s) [noise=true , address=45.91.116.9:4274 , piece_id=XKNACYYGKETXPQTRTATURZ2RRXLIDNUJOG2EKLG46OQUSMN3K33Q, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: storage node overloaded, request limit: 2000
Jul 9 01:08:10.737498024 (375h57m45.508139359s) [noise=true , address=104.36.82.39:4726 , piece_id=5OLQ7AGZ5I7AOKBCSP34VPYIFYB2IVWZJ6WWA4DTDJ5LRNXICUSQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:08:54.804118607 (375h58m29.574759942s) [noise=true , address=104.36.82.18:4390 , piece_id=G24NDRJ3NCKCWG6OFNUVRXE6HRUDZ7I4P6AKQ446XBKHNIOCD2TQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12AZCJHCR7rF4r62Bg7NcBkpNoLb6r6rEN2a5hZ9KNxJTtjV8jp, address:104.36.82.18:4390): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804258494 (375h58m29.574899819s) [noise=true , address=104.36.82.23:4466 , piece_id=TTUJEA6JYTHG3WPYE5VLFPS6IYRXBUI5IZV3FV5725A6XPMJVZLQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12rkNnk9RUhKBJ3EsakqMRb3wYwrhRLhB4ZYbrdfKPDk8erRJpP, address:104.36.82.23:4466): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804300485 (375h58m29.57494184s) [noise=true , address=104.36.82.17:4350 , piece_id=YDPD4LOKKFCGXAGBJGNL7D77APW27OHIYSWYUR7T6XX4ZFP3FHZA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804343667 (375h58m29.574985002s) [noise=true , address=104.36.82.3:4786 , piece_id=24V2D5IBAXBHUDVZ7KJXDNZHVHC2H6IWQ3NAISWJ5PZSPF4TOVWA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:127aTG7tofhLA5799hT6gvZZSkT1vvWWFejx7DS4Zcz8mBHHsuE, address:104.36.82.3:4786): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804418601 (375h58m29.575059936s) [noise=true , address=104.36.82.19:4411 , piece_id=CURSEVNLHJHEMVYVLGYHIVTHMYSZSEFLCR5ZDMFSQDREHNBLKSUA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12nradx4SMYLkuJw3Ad6o9Sq1iNtQR6GDu8qg7tCzZzNcTjVfyd, address:104.36.82.19:4411): protocol: write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection; write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection
Jul 9 01:08:54.804449420 (375h58m29.575090755s) [seg_pos={0 0}, upload=300780]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 9 01:08:55.034441116 (375h58m29.805082451s) [upload=300780]: Done uploading segments...
Jul 9 01:08:55.034460744 (375h58m29.805102079s) [upload=300780]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 25.249635742s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [X.X...XX....X.....X...X......XX....X...X.X....X...X.X..X...X] duration:
30.000701489s
FAIL (2025-07-09 01:25:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000720796s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..............................................X......XX.XX..] duration:
58.051872848s
FAIL (2025-07-09 01:25:15 UTC)
orbiter warning: context deadline exceeded
Duration: 58.051891393s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................X........................X....] duration:
5.779886923s
FAIL (2025-07-09 01:25:09 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ommit
Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments...
Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil>
Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload
Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload
Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments...
Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false
Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment...
Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment.
Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment
Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments...
Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection
Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection
Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection
Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments...
Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.779908373s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
30.315392691s
FAIL (2025-07-09 01:25:09 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
7:55.442899027 (375h57m30.213540362s) [upload=300780]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:07:55.885104488 (375h57m30.655745823s) [seg_pos={0 0}, upload=300780]: Begin upload segment...
Jul 9 01:07:55.885150927 (375h57m30.655792262s) [seg_pos={0 0}, upload=300780]: Done begin upload segment.
Jul 9 01:07:55.885170996 (375h57m30.655812331s) [noise=true , address=77.90.31.14:4669 , piece_id=KQNMOTC4GWOGYIZY2QK27LKEPN225LAYFJZWCJC6MRCK7QBPRFCQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885528510 (375h57m30.656169845s) [noise=true , address=104.36.82.23:4466 , piece_id=TTUJEA6JYTHG3WPYE5VLFPS6IYRXBUI5IZV3FV5725A6XPMJVZLQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885790180 (375h57m30.656431515s) [noise=true , address=77.90.31.13:4649 , piece_id=HXJ735B5UBJP4VSLABNV2RDSI6XELDRJ7DPXJXOZHU7QPGY6ON3Q, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885880773 (375h57m30.656522108s) [noise=true , address=77.90.31.16:4755 , piece_id=XAPAI7EP2B7XCJI6OBUVPPTPVAHARZVNVGUP4WBMLFPQNBOXUMBQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886081398 (375h57m30.656722733s) [noise=true , address=77.90.31.17:4801 , piece_id=K2UIXAWUISIA5KCKQCZMBJ3BHB2MS44MVMB7T4T2CCA2PMGJJDDA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886215583 (375h57m30.656856919s) [noise=true , address=104.36.82.39:4726 , piece_id=5OLQ7AGZ5I7AOKBCSP34VPYIFYB2IVWZJ6WWA4DTDJ5LRNXICUSQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886404385 (375h57m30.65704572s) [noise=true , address=45.91.116.9:4274 , piece_id=XKNACYYGKETXPQTRTATURZ2RRXLIDNUJOG2EKLG46OQUSMN3K33Q, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886513444 (375h57m30.657154769s) [noise=true , address=77.90.31.19:4861 , piece_id=RVU6MOFXHFWCAIKRNXKGQT4KADZSLIXXNIG6NDHHNJZ7KEBAXICA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886808428 (375h57m30.657449763s) [noise=true , address=104.36.82.3:4786 , piece_id=24V2D5IBAXBHUDVZ7KJXDNZHVHC2H6IWQ3NAISWJ5PZSPF4TOVWA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.887107310 (375h57m30.657748645s) [noise=true , address=104.36.82.17:4350 , piece_id=YDPD4LOKKFCGXAGBJGNL7D77APW27OHIYSWYUR7T6XX4ZFP3FHZA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.887393648 (375h57m30.658034983s) [noise=true , address=156.146.43.225:4439 , piece_id=WHIPMS22HXTUQAOUPNXT2DLA6XL2YWXM43KLEC55YFEY2SKBNIBA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888027160 (375h57m30.658668495s) [noise=true , address=104.36.82.18:4390 , piece_id=G24NDRJ3NCKCWG6OFNUVRXE6HRUDZ7I4P6AKQ446XBKHNIOCD2TQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888272730 (375h57m30.658914065s) [noise=true , address=77.90.31.12:4622 , piece_id=F566GJ7G6WUKJE63V7Z25TOJ25LDGSVYHPBYMQCQQLRDTRYBJFYQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888495466 (375h57m30.659136801s) [noise=true , address=77.90.31.13:4656 , piece_id=BM62XMC4FBDFBSF7BRQAUE2UWJZU5JUODULKP7CWVPRXFFCGS2CQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888749441 (375h57m30.659390776s) [noise=true , address=104.36.82.19:4411 , piece_id=CURSEVNLHJHEMVYVLGYHIVTHMYSZSEFLCR5ZDMFSQDREHNBLKSUA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889008307 (375h57m30.659649642s) [noise=true , address=77.90.31.11:4557 , piece_id=IPHAW23VXAVZ4XP3TOMX2RMVO2JDC65HLRRNBSMZTA5JJXY5EABQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889236964 (375h57m30.659878299s) [noise=true , address=77.90.31.18:4840 , piece_id=SNZW25XL4VJ7UN5ZMGU4ZSOW6FQA5ZEUPQCV5OKRSPQIA3F4XGSA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889458348 (375h57m30.660099683s) [noise=true , address=77.90.31.14:4677 , piece_id=53ZLF5H2KYVI6HM3GSHVC6JKGJ7TV3624DFNCXCK76ZIM5YVKLSQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:56.955856205 (375h57m31.72649754s) [noise=true , address=156.146.43.225:4439 , piece_id=WHIPMS22HXTUQAOUPNXT2DLA6XL2YWXM43KLEC55YFEY2SKBNIBA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12VztoHQmCpGLhyN8wKXqJ9u7BzG9UbbeQinmquUZ9Rb2e2w9Ze, address:156.146.43.225:4439): protocol: storage node overloaded, request limit: 2000
Jul 9 01:07:57.480276305 (375h57m32.25091764s) [upload=300780]: Next returned nil segment
Jul 9 01:07:57.480291304 (375h57m32.250932639s) [upload=300780]: Waiting for error group managing segments...
Jul 9 01:07:57.892917917 (375h57m32.663559252s) [noise=true , address=77.90.31.13:4649 , piece_id=HXJ735B5UBJP4VSLABNV2RDSI6XELDRJ7DPXJXOZHU7QPGY6ON3Q, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.894965904 (375h57m32.66560724s) [noise=true , address=77.90.31.19:4861 , piece_id=RVU6MOFXHFWCAIKRNXKGQT4KADZSLIXXNIG6NDHHNJZ7KEBAXICA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.895386990 (375h57m32.666028325s) [noise=true , address=77.90.31.13:4656 , piece_id=BM62XMC4FBDFBSF7BRQAUE2UWJZU5JUODULKP7CWVPRXFFCGS2CQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.897322493 (375h57m32.667963828s) [noise=true , address=77.90.31.11:4557 , piece_id=IPHAW23VXAVZ4XP3TOMX2RMVO2JDC65HLRRNBSMZTA5JJXY5EABQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.899196428 (375h57m32.669837753s) [noise=true , address=77.90.31.18:4840 , piece_id=SNZW25XL4VJ7UN5ZMGU4ZSOW6FQA5ZEUPQCV5OKRSPQIA3F4XGSA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.901992567 (375h57m32.672633902s) [noise=true , address=77.90.31.14:4677 , piece_id=53ZLF5H2KYVI6HM3GSHVC6JKGJ7TV3624DFNCXCK76ZIM5YVKLSQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.907178669 (375h57m32.677820014s) [noise=true , address=77.90.31.12:4622 , piece_id=F566GJ7G6WUKJE63V7Z25TOJ25LDGSVYHPBYMQCQQLRDTRYBJFYQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.917659831 (375h57m32.688301166s) [noise=true , address=77.90.31.14:4669 , piece_id=KQNMOTC4GWOGYIZY2QK27LKEPN225LAYFJZWCJC6MRCK7QBPRFCQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.928100055 (375h57m32.69874139s) [noise=true , address=77.90.31.17:4801 , piece_id=K2UIXAWUISIA5KCKQCZMBJ3BHB2MS44MVMB7T4T2CCA2PMGJJDDA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.955811390 (375h57m32.726452725s) [noise=true , address=77.90.31.16:4755 , piece_id=XAPAI7EP2B7XCJI6OBUVPPTPVAHARZVNVGUP4WBMLFPQNBOXUMBQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:08:00.166279605 (375h57m34.93692094s) [noise=true , address=45.91.116.9:4274 , piece_id=XKNACYYGKETXPQTRTATURZ2RRXLIDNUJOG2EKLG46OQUSMN3K33Q, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: storage node overloaded, request limit: 2000
Jul 9 01:08:10.737498024 (375h57m45.508139359s) [noise=true , address=104.36.82.39:4726 , piece_id=5OLQ7AGZ5I7AOKBCSP34VPYIFYB2IVWZJ6WWA4DTDJ5LRNXICUSQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:08:54.804118607 (375h58m29.574759942s) [noise=true , address=104.36.82.18:4390 , piece_id=G24NDRJ3NCKCWG6OFNUVRXE6HRUDZ7I4P6AKQ446XBKHNIOCD2TQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12AZCJHCR7rF4r62Bg7NcBkpNoLb6r6rEN2a5hZ9KNxJTtjV8jp, address:104.36.82.18:4390): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804258494 (375h58m29.574899819s) [noise=true , address=104.36.82.23:4466 , piece_id=TTUJEA6JYTHG3WPYE5VLFPS6IYRXBUI5IZV3FV5725A6XPMJVZLQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12rkNnk9RUhKBJ3EsakqMRb3wYwrhRLhB4ZYbrdfKPDk8erRJpP, address:104.36.82.23:4466): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804300485 (375h58m29.57494184s) [noise=true , address=104.36.82.17:4350 , piece_id=YDPD4LOKKFCGXAGBJGNL7D77APW27OHIYSWYUR7T6XX4ZFP3FHZA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804343667 (375h58m29.574985002s) [noise=true , address=104.36.82.3:4786 , piece_id=24V2D5IBAXBHUDVZ7KJXDNZHVHC2H6IWQ3NAISWJ5PZSPF4TOVWA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:127aTG7tofhLA5799hT6gvZZSkT1vvWWFejx7DS4Zcz8mBHHsuE, address:104.36.82.3:4786): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804418601 (375h58m29.575059936s) [noise=true , address=104.36.82.19:4411 , piece_id=CURSEVNLHJHEMVYVLGYHIVTHMYSZSEFLCR5ZDMFSQDREHNBLKSUA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12nradx4SMYLkuJw3Ad6o9Sq1iNtQR6GDu8qg7tCzZzNcTjVfyd, address:104.36.82.19:4411): protocol: write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection; write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection
Jul 9 01:08:54.804449420 (375h58m29.575090755s) [seg_pos={0 0}, upload=300780]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 9 01:08:55.034441116 (375h58m29.805082451s) [upload=300780]: Done uploading segments...
Jul 9 01:08:55.034460744 (375h58m29.805102079s) [upload=300780]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 30.31540725s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................X........................X...] duration:
24.964246776s
FAIL (2025-07-09 01:24:52 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ommit
Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments...
Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil>
Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload
Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload
Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments...
Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false
Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment...
Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment.
Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment
Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments...
Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection
Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection
Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection
Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments...
Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 24.96427461s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
56.659848668s
FAIL (2025-07-09 01:24:38 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
7:55.442899027 (375h57m30.213540362s) [upload=300780]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:07:55.885104488 (375h57m30.655745823s) [seg_pos={0 0}, upload=300780]: Begin upload segment...
Jul 9 01:07:55.885150927 (375h57m30.655792262s) [seg_pos={0 0}, upload=300780]: Done begin upload segment.
Jul 9 01:07:55.885170996 (375h57m30.655812331s) [noise=true , address=77.90.31.14:4669 , piece_id=KQNMOTC4GWOGYIZY2QK27LKEPN225LAYFJZWCJC6MRCK7QBPRFCQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885528510 (375h57m30.656169845s) [noise=true , address=104.36.82.23:4466 , piece_id=TTUJEA6JYTHG3WPYE5VLFPS6IYRXBUI5IZV3FV5725A6XPMJVZLQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885790180 (375h57m30.656431515s) [noise=true , address=77.90.31.13:4649 , piece_id=HXJ735B5UBJP4VSLABNV2RDSI6XELDRJ7DPXJXOZHU7QPGY6ON3Q, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885880773 (375h57m30.656522108s) [noise=true , address=77.90.31.16:4755 , piece_id=XAPAI7EP2B7XCJI6OBUVPPTPVAHARZVNVGUP4WBMLFPQNBOXUMBQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886081398 (375h57m30.656722733s) [noise=true , address=77.90.31.17:4801 , piece_id=K2UIXAWUISIA5KCKQCZMBJ3BHB2MS44MVMB7T4T2CCA2PMGJJDDA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886215583 (375h57m30.656856919s) [noise=true , address=104.36.82.39:4726 , piece_id=5OLQ7AGZ5I7AOKBCSP34VPYIFYB2IVWZJ6WWA4DTDJ5LRNXICUSQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886404385 (375h57m30.65704572s) [noise=true , address=45.91.116.9:4274 , piece_id=XKNACYYGKETXPQTRTATURZ2RRXLIDNUJOG2EKLG46OQUSMN3K33Q, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886513444 (375h57m30.657154769s) [noise=true , address=77.90.31.19:4861 , piece_id=RVU6MOFXHFWCAIKRNXKGQT4KADZSLIXXNIG6NDHHNJZ7KEBAXICA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886808428 (375h57m30.657449763s) [noise=true , address=104.36.82.3:4786 , piece_id=24V2D5IBAXBHUDVZ7KJXDNZHVHC2H6IWQ3NAISWJ5PZSPF4TOVWA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.887107310 (375h57m30.657748645s) [noise=true , address=104.36.82.17:4350 , piece_id=YDPD4LOKKFCGXAGBJGNL7D77APW27OHIYSWYUR7T6XX4ZFP3FHZA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.887393648 (375h57m30.658034983s) [noise=true , address=156.146.43.225:4439 , piece_id=WHIPMS22HXTUQAOUPNXT2DLA6XL2YWXM43KLEC55YFEY2SKBNIBA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888027160 (375h57m30.658668495s) [noise=true , address=104.36.82.18:4390 , piece_id=G24NDRJ3NCKCWG6OFNUVRXE6HRUDZ7I4P6AKQ446XBKHNIOCD2TQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888272730 (375h57m30.658914065s) [noise=true , address=77.90.31.12:4622 , piece_id=F566GJ7G6WUKJE63V7Z25TOJ25LDGSVYHPBYMQCQQLRDTRYBJFYQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888495466 (375h57m30.659136801s) [noise=true , address=77.90.31.13:4656 , piece_id=BM62XMC4FBDFBSF7BRQAUE2UWJZU5JUODULKP7CWVPRXFFCGS2CQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888749441 (375h57m30.659390776s) [noise=true , address=104.36.82.19:4411 , piece_id=CURSEVNLHJHEMVYVLGYHIVTHMYSZSEFLCR5ZDMFSQDREHNBLKSUA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889008307 (375h57m30.659649642s) [noise=true , address=77.90.31.11:4557 , piece_id=IPHAW23VXAVZ4XP3TOMX2RMVO2JDC65HLRRNBSMZTA5JJXY5EABQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889236964 (375h57m30.659878299s) [noise=true , address=77.90.31.18:4840 , piece_id=SNZW25XL4VJ7UN5ZMGU4ZSOW6FQA5ZEUPQCV5OKRSPQIA3F4XGSA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889458348 (375h57m30.660099683s) [noise=true , address=77.90.31.14:4677 , piece_id=53ZLF5H2KYVI6HM3GSHVC6JKGJ7TV3624DFNCXCK76ZIM5YVKLSQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:56.955856205 (375h57m31.72649754s) [noise=true , address=156.146.43.225:4439 , piece_id=WHIPMS22HXTUQAOUPNXT2DLA6XL2YWXM43KLEC55YFEY2SKBNIBA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12VztoHQmCpGLhyN8wKXqJ9u7BzG9UbbeQinmquUZ9Rb2e2w9Ze, address:156.146.43.225:4439): protocol: storage node overloaded, request limit: 2000
Jul 9 01:07:57.480276305 (375h57m32.25091764s) [upload=300780]: Next returned nil segment
Jul 9 01:07:57.480291304 (375h57m32.250932639s) [upload=300780]: Waiting for error group managing segments...
Jul 9 01:07:57.892917917 (375h57m32.663559252s) [noise=true , address=77.90.31.13:4649 , piece_id=HXJ735B5UBJP4VSLABNV2RDSI6XELDRJ7DPXJXOZHU7QPGY6ON3Q, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.894965904 (375h57m32.66560724s) [noise=true , address=77.90.31.19:4861 , piece_id=RVU6MOFXHFWCAIKRNXKGQT4KADZSLIXXNIG6NDHHNJZ7KEBAXICA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.895386990 (375h57m32.666028325s) [noise=true , address=77.90.31.13:4656 , piece_id=BM62XMC4FBDFBSF7BRQAUE2UWJZU5JUODULKP7CWVPRXFFCGS2CQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.897322493 (375h57m32.667963828s) [noise=true , address=77.90.31.11:4557 , piece_id=IPHAW23VXAVZ4XP3TOMX2RMVO2JDC65HLRRNBSMZTA5JJXY5EABQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.899196428 (375h57m32.669837753s) [noise=true , address=77.90.31.18:4840 , piece_id=SNZW25XL4VJ7UN5ZMGU4ZSOW6FQA5ZEUPQCV5OKRSPQIA3F4XGSA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.901992567 (375h57m32.672633902s) [noise=true , address=77.90.31.14:4677 , piece_id=53ZLF5H2KYVI6HM3GSHVC6JKGJ7TV3624DFNCXCK76ZIM5YVKLSQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.907178669 (375h57m32.677820014s) [noise=true , address=77.90.31.12:4622 , piece_id=F566GJ7G6WUKJE63V7Z25TOJ25LDGSVYHPBYMQCQQLRDTRYBJFYQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.917659831 (375h57m32.688301166s) [noise=true , address=77.90.31.14:4669 , piece_id=KQNMOTC4GWOGYIZY2QK27LKEPN225LAYFJZWCJC6MRCK7QBPRFCQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.928100055 (375h57m32.69874139s) [noise=true , address=77.90.31.17:4801 , piece_id=K2UIXAWUISIA5KCKQCZMBJ3BHB2MS44MVMB7T4T2CCA2PMGJJDDA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.955811390 (375h57m32.726452725s) [noise=true , address=77.90.31.16:4755 , piece_id=XAPAI7EP2B7XCJI6OBUVPPTPVAHARZVNVGUP4WBMLFPQNBOXUMBQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:08:00.166279605 (375h57m34.93692094s) [noise=true , address=45.91.116.9:4274 , piece_id=XKNACYYGKETXPQTRTATURZ2RRXLIDNUJOG2EKLG46OQUSMN3K33Q, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: storage node overloaded, request limit: 2000
Jul 9 01:08:10.737498024 (375h57m45.508139359s) [noise=true , address=104.36.82.39:4726 , piece_id=5OLQ7AGZ5I7AOKBCSP34VPYIFYB2IVWZJ6WWA4DTDJ5LRNXICUSQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:08:54.804118607 (375h58m29.574759942s) [noise=true , address=104.36.82.18:4390 , piece_id=G24NDRJ3NCKCWG6OFNUVRXE6HRUDZ7I4P6AKQ446XBKHNIOCD2TQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12AZCJHCR7rF4r62Bg7NcBkpNoLb6r6rEN2a5hZ9KNxJTtjV8jp, address:104.36.82.18:4390): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804258494 (375h58m29.574899819s) [noise=true , address=104.36.82.23:4466 , piece_id=TTUJEA6JYTHG3WPYE5VLFPS6IYRXBUI5IZV3FV5725A6XPMJVZLQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12rkNnk9RUhKBJ3EsakqMRb3wYwrhRLhB4ZYbrdfKPDk8erRJpP, address:104.36.82.23:4466): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804300485 (375h58m29.57494184s) [noise=true , address=104.36.82.17:4350 , piece_id=YDPD4LOKKFCGXAGBJGNL7D77APW27OHIYSWYUR7T6XX4ZFP3FHZA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804343667 (375h58m29.574985002s) [noise=true , address=104.36.82.3:4786 , piece_id=24V2D5IBAXBHUDVZ7KJXDNZHVHC2H6IWQ3NAISWJ5PZSPF4TOVWA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:127aTG7tofhLA5799hT6gvZZSkT1vvWWFejx7DS4Zcz8mBHHsuE, address:104.36.82.3:4786): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804418601 (375h58m29.575059936s) [noise=true , address=104.36.82.19:4411 , piece_id=CURSEVNLHJHEMVYVLGYHIVTHMYSZSEFLCR5ZDMFSQDREHNBLKSUA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12nradx4SMYLkuJw3Ad6o9Sq1iNtQR6GDu8qg7tCzZzNcTjVfyd, address:104.36.82.19:4411): protocol: write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection; write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection
Jul 9 01:08:54.804449420 (375h58m29.575090755s) [seg_pos={0 0}, upload=300780]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 9 01:08:55.034441116 (375h58m29.805082451s) [upload=300780]: Done uploading segments...
Jul 9 01:08:55.034460744 (375h58m29.805102079s) [upload=300780]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 56.65986516s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.X.X...XX....X.....X...X......XX....X...X.X....X...X.X..X...] duration:
6.475171956s
FAIL (2025-07-09 01:24:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.475195592s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................X........................X..] duration:
17.438164974s
FAIL (2025-07-09 01:24:18 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ommit
Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments...
Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil>
Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload
Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload
Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments...
Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false
Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment...
Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment.
Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment
Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments...
Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection
Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection
Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection
Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments...
Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 17.438195001s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [..X.X...XX....X.....X...X......XX....X...X.X....X...X.X..X..] duration:
5.910342211s
FAIL (2025-07-09 01:24:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.910367972s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X................................X........................X.] duration:
5.711605421s
FAIL (2025-07-09 01:23:51 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ommit
Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments...
Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil>
Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload
Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload
Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments...
Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false
Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment...
Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment.
Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment
Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments...
Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection
Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection
Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection
Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments...
Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.711643282s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [X..X.X...XX....X.....X...X......XX....X...X.X....X...X.X..X.] duration:
17.184348669s
FAIL (2025-07-09 01:23:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 17.184371183s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...............................................X......XX.XX.] duration:
33.654045681s
FAIL (2025-07-09 01:23:43 UTC)
orbiter warning: context deadline exceeded
Duration: 33.654061722s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
22.577771783s
FAIL (2025-07-09 01:23:41 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
7:55.442899027 (375h57m30.213540362s) [upload=300780]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:07:55.885104488 (375h57m30.655745823s) [seg_pos={0 0}, upload=300780]: Begin upload segment...
Jul 9 01:07:55.885150927 (375h57m30.655792262s) [seg_pos={0 0}, upload=300780]: Done begin upload segment.
Jul 9 01:07:55.885170996 (375h57m30.655812331s) [noise=true , address=77.90.31.14:4669 , piece_id=KQNMOTC4GWOGYIZY2QK27LKEPN225LAYFJZWCJC6MRCK7QBPRFCQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885528510 (375h57m30.656169845s) [noise=true , address=104.36.82.23:4466 , piece_id=TTUJEA6JYTHG3WPYE5VLFPS6IYRXBUI5IZV3FV5725A6XPMJVZLQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885790180 (375h57m30.656431515s) [noise=true , address=77.90.31.13:4649 , piece_id=HXJ735B5UBJP4VSLABNV2RDSI6XELDRJ7DPXJXOZHU7QPGY6ON3Q, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885880773 (375h57m30.656522108s) [noise=true , address=77.90.31.16:4755 , piece_id=XAPAI7EP2B7XCJI6OBUVPPTPVAHARZVNVGUP4WBMLFPQNBOXUMBQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886081398 (375h57m30.656722733s) [noise=true , address=77.90.31.17:4801 , piece_id=K2UIXAWUISIA5KCKQCZMBJ3BHB2MS44MVMB7T4T2CCA2PMGJJDDA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886215583 (375h57m30.656856919s) [noise=true , address=104.36.82.39:4726 , piece_id=5OLQ7AGZ5I7AOKBCSP34VPYIFYB2IVWZJ6WWA4DTDJ5LRNXICUSQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886404385 (375h57m30.65704572s) [noise=true , address=45.91.116.9:4274 , piece_id=XKNACYYGKETXPQTRTATURZ2RRXLIDNUJOG2EKLG46OQUSMN3K33Q, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886513444 (375h57m30.657154769s) [noise=true , address=77.90.31.19:4861 , piece_id=RVU6MOFXHFWCAIKRNXKGQT4KADZSLIXXNIG6NDHHNJZ7KEBAXICA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886808428 (375h57m30.657449763s) [noise=true , address=104.36.82.3:4786 , piece_id=24V2D5IBAXBHUDVZ7KJXDNZHVHC2H6IWQ3NAISWJ5PZSPF4TOVWA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.887107310 (375h57m30.657748645s) [noise=true , address=104.36.82.17:4350 , piece_id=YDPD4LOKKFCGXAGBJGNL7D77APW27OHIYSWYUR7T6XX4ZFP3FHZA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.887393648 (375h57m30.658034983s) [noise=true , address=156.146.43.225:4439 , piece_id=WHIPMS22HXTUQAOUPNXT2DLA6XL2YWXM43KLEC55YFEY2SKBNIBA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888027160 (375h57m30.658668495s) [noise=true , address=104.36.82.18:4390 , piece_id=G24NDRJ3NCKCWG6OFNUVRXE6HRUDZ7I4P6AKQ446XBKHNIOCD2TQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888272730 (375h57m30.658914065s) [noise=true , address=77.90.31.12:4622 , piece_id=F566GJ7G6WUKJE63V7Z25TOJ25LDGSVYHPBYMQCQQLRDTRYBJFYQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888495466 (375h57m30.659136801s) [noise=true , address=77.90.31.13:4656 , piece_id=BM62XMC4FBDFBSF7BRQAUE2UWJZU5JUODULKP7CWVPRXFFCGS2CQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888749441 (375h57m30.659390776s) [noise=true , address=104.36.82.19:4411 , piece_id=CURSEVNLHJHEMVYVLGYHIVTHMYSZSEFLCR5ZDMFSQDREHNBLKSUA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889008307 (375h57m30.659649642s) [noise=true , address=77.90.31.11:4557 , piece_id=IPHAW23VXAVZ4XP3TOMX2RMVO2JDC65HLRRNBSMZTA5JJXY5EABQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889236964 (375h57m30.659878299s) [noise=true , address=77.90.31.18:4840 , piece_id=SNZW25XL4VJ7UN5ZMGU4ZSOW6FQA5ZEUPQCV5OKRSPQIA3F4XGSA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889458348 (375h57m30.660099683s) [noise=true , address=77.90.31.14:4677 , piece_id=53ZLF5H2KYVI6HM3GSHVC6JKGJ7TV3624DFNCXCK76ZIM5YVKLSQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:56.955856205 (375h57m31.72649754s) [noise=true , address=156.146.43.225:4439 , piece_id=WHIPMS22HXTUQAOUPNXT2DLA6XL2YWXM43KLEC55YFEY2SKBNIBA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12VztoHQmCpGLhyN8wKXqJ9u7BzG9UbbeQinmquUZ9Rb2e2w9Ze, address:156.146.43.225:4439): protocol: storage node overloaded, request limit: 2000
Jul 9 01:07:57.480276305 (375h57m32.25091764s) [upload=300780]: Next returned nil segment
Jul 9 01:07:57.480291304 (375h57m32.250932639s) [upload=300780]: Waiting for error group managing segments...
Jul 9 01:07:57.892917917 (375h57m32.663559252s) [noise=true , address=77.90.31.13:4649 , piece_id=HXJ735B5UBJP4VSLABNV2RDSI6XELDRJ7DPXJXOZHU7QPGY6ON3Q, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.894965904 (375h57m32.66560724s) [noise=true , address=77.90.31.19:4861 , piece_id=RVU6MOFXHFWCAIKRNXKGQT4KADZSLIXXNIG6NDHHNJZ7KEBAXICA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.895386990 (375h57m32.666028325s) [noise=true , address=77.90.31.13:4656 , piece_id=BM62XMC4FBDFBSF7BRQAUE2UWJZU5JUODULKP7CWVPRXFFCGS2CQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.897322493 (375h57m32.667963828s) [noise=true , address=77.90.31.11:4557 , piece_id=IPHAW23VXAVZ4XP3TOMX2RMVO2JDC65HLRRNBSMZTA5JJXY5EABQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.899196428 (375h57m32.669837753s) [noise=true , address=77.90.31.18:4840 , piece_id=SNZW25XL4VJ7UN5ZMGU4ZSOW6FQA5ZEUPQCV5OKRSPQIA3F4XGSA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.901992567 (375h57m32.672633902s) [noise=true , address=77.90.31.14:4677 , piece_id=53ZLF5H2KYVI6HM3GSHVC6JKGJ7TV3624DFNCXCK76ZIM5YVKLSQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.907178669 (375h57m32.677820014s) [noise=true , address=77.90.31.12:4622 , piece_id=F566GJ7G6WUKJE63V7Z25TOJ25LDGSVYHPBYMQCQQLRDTRYBJFYQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.917659831 (375h57m32.688301166s) [noise=true , address=77.90.31.14:4669 , piece_id=KQNMOTC4GWOGYIZY2QK27LKEPN225LAYFJZWCJC6MRCK7QBPRFCQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.928100055 (375h57m32.69874139s) [noise=true , address=77.90.31.17:4801 , piece_id=K2UIXAWUISIA5KCKQCZMBJ3BHB2MS44MVMB7T4T2CCA2PMGJJDDA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.955811390 (375h57m32.726452725s) [noise=true , address=77.90.31.16:4755 , piece_id=XAPAI7EP2B7XCJI6OBUVPPTPVAHARZVNVGUP4WBMLFPQNBOXUMBQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:08:00.166279605 (375h57m34.93692094s) [noise=true , address=45.91.116.9:4274 , piece_id=XKNACYYGKETXPQTRTATURZ2RRXLIDNUJOG2EKLG46OQUSMN3K33Q, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: storage node overloaded, request limit: 2000
Jul 9 01:08:10.737498024 (375h57m45.508139359s) [noise=true , address=104.36.82.39:4726 , piece_id=5OLQ7AGZ5I7AOKBCSP34VPYIFYB2IVWZJ6WWA4DTDJ5LRNXICUSQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:08:54.804118607 (375h58m29.574759942s) [noise=true , address=104.36.82.18:4390 , piece_id=G24NDRJ3NCKCWG6OFNUVRXE6HRUDZ7I4P6AKQ446XBKHNIOCD2TQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12AZCJHCR7rF4r62Bg7NcBkpNoLb6r6rEN2a5hZ9KNxJTtjV8jp, address:104.36.82.18:4390): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804258494 (375h58m29.574899819s) [noise=true , address=104.36.82.23:4466 , piece_id=TTUJEA6JYTHG3WPYE5VLFPS6IYRXBUI5IZV3FV5725A6XPMJVZLQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12rkNnk9RUhKBJ3EsakqMRb3wYwrhRLhB4ZYbrdfKPDk8erRJpP, address:104.36.82.23:4466): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804300485 (375h58m29.57494184s) [noise=true , address=104.36.82.17:4350 , piece_id=YDPD4LOKKFCGXAGBJGNL7D77APW27OHIYSWYUR7T6XX4ZFP3FHZA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804343667 (375h58m29.574985002s) [noise=true , address=104.36.82.3:4786 , piece_id=24V2D5IBAXBHUDVZ7KJXDNZHVHC2H6IWQ3NAISWJ5PZSPF4TOVWA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:127aTG7tofhLA5799hT6gvZZSkT1vvWWFejx7DS4Zcz8mBHHsuE, address:104.36.82.3:4786): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804418601 (375h58m29.575059936s) [noise=true , address=104.36.82.19:4411 , piece_id=CURSEVNLHJHEMVYVLGYHIVTHMYSZSEFLCR5ZDMFSQDREHNBLKSUA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12nradx4SMYLkuJw3Ad6o9Sq1iNtQR6GDu8qg7tCzZzNcTjVfyd, address:104.36.82.19:4411): protocol: write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection; write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection
Jul 9 01:08:54.804449420 (375h58m29.575090755s) [seg_pos={0 0}, upload=300780]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 9 01:08:55.034441116 (375h58m29.805082451s) [upload=300780]: Done uploading segments...
Jul 9 01:08:55.034460744 (375h58m29.805102079s) [upload=300780]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 22.577806129s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X................................X........................X] duration:
30.236907271s
FAIL (2025-07-09 01:23:30 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ommit
Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments...
Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil>
Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload
Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload
Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments...
Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false
Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment...
Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment.
Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece...
Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment
Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments...
Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil>
Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection
Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection
Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection
Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments...
Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.23695385s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [X...............................................X......XX.XX] duration:
1m0.000877282s
FAIL (2025-07-09 01:23:09 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.000890556s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [.X..X.X...XX....X.....X...X......XX....X...X.X....X...X.X..X] duration:
30.001313333s
FAIL (2025-07-09 01:22:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001352799s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
28.951484108s
FAIL (2025-07-09 01:22:52 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
7:55.442899027 (375h57m30.213540362s) [upload=300780]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:07:55.885104488 (375h57m30.655745823s) [seg_pos={0 0}, upload=300780]: Begin upload segment...
Jul 9 01:07:55.885150927 (375h57m30.655792262s) [seg_pos={0 0}, upload=300780]: Done begin upload segment.
Jul 9 01:07:55.885170996 (375h57m30.655812331s) [noise=true , address=77.90.31.14:4669 , piece_id=KQNMOTC4GWOGYIZY2QK27LKEPN225LAYFJZWCJC6MRCK7QBPRFCQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885528510 (375h57m30.656169845s) [noise=true , address=104.36.82.23:4466 , piece_id=TTUJEA6JYTHG3WPYE5VLFPS6IYRXBUI5IZV3FV5725A6XPMJVZLQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885790180 (375h57m30.656431515s) [noise=true , address=77.90.31.13:4649 , piece_id=HXJ735B5UBJP4VSLABNV2RDSI6XELDRJ7DPXJXOZHU7QPGY6ON3Q, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.885880773 (375h57m30.656522108s) [noise=true , address=77.90.31.16:4755 , piece_id=XAPAI7EP2B7XCJI6OBUVPPTPVAHARZVNVGUP4WBMLFPQNBOXUMBQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886081398 (375h57m30.656722733s) [noise=true , address=77.90.31.17:4801 , piece_id=K2UIXAWUISIA5KCKQCZMBJ3BHB2MS44MVMB7T4T2CCA2PMGJJDDA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886215583 (375h57m30.656856919s) [noise=true , address=104.36.82.39:4726 , piece_id=5OLQ7AGZ5I7AOKBCSP34VPYIFYB2IVWZJ6WWA4DTDJ5LRNXICUSQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886404385 (375h57m30.65704572s) [noise=true , address=45.91.116.9:4274 , piece_id=XKNACYYGKETXPQTRTATURZ2RRXLIDNUJOG2EKLG46OQUSMN3K33Q, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886513444 (375h57m30.657154769s) [noise=true , address=77.90.31.19:4861 , piece_id=RVU6MOFXHFWCAIKRNXKGQT4KADZSLIXXNIG6NDHHNJZ7KEBAXICA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.886808428 (375h57m30.657449763s) [noise=true , address=104.36.82.3:4786 , piece_id=24V2D5IBAXBHUDVZ7KJXDNZHVHC2H6IWQ3NAISWJ5PZSPF4TOVWA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.887107310 (375h57m30.657748645s) [noise=true , address=104.36.82.17:4350 , piece_id=YDPD4LOKKFCGXAGBJGNL7D77APW27OHIYSWYUR7T6XX4ZFP3FHZA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.887393648 (375h57m30.658034983s) [noise=true , address=156.146.43.225:4439 , piece_id=WHIPMS22HXTUQAOUPNXT2DLA6XL2YWXM43KLEC55YFEY2SKBNIBA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888027160 (375h57m30.658668495s) [noise=true , address=104.36.82.18:4390 , piece_id=G24NDRJ3NCKCWG6OFNUVRXE6HRUDZ7I4P6AKQ446XBKHNIOCD2TQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888272730 (375h57m30.658914065s) [noise=true , address=77.90.31.12:4622 , piece_id=F566GJ7G6WUKJE63V7Z25TOJ25LDGSVYHPBYMQCQQLRDTRYBJFYQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888495466 (375h57m30.659136801s) [noise=true , address=77.90.31.13:4656 , piece_id=BM62XMC4FBDFBSF7BRQAUE2UWJZU5JUODULKP7CWVPRXFFCGS2CQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.888749441 (375h57m30.659390776s) [noise=true , address=104.36.82.19:4411 , piece_id=CURSEVNLHJHEMVYVLGYHIVTHMYSZSEFLCR5ZDMFSQDREHNBLKSUA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889008307 (375h57m30.659649642s) [noise=true , address=77.90.31.11:4557 , piece_id=IPHAW23VXAVZ4XP3TOMX2RMVO2JDC65HLRRNBSMZTA5JJXY5EABQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889236964 (375h57m30.659878299s) [noise=true , address=77.90.31.18:4840 , piece_id=SNZW25XL4VJ7UN5ZMGU4ZSOW6FQA5ZEUPQCV5OKRSPQIA3F4XGSA, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:55.889458348 (375h57m30.660099683s) [noise=true , address=77.90.31.14:4677 , piece_id=53ZLF5H2KYVI6HM3GSHVC6JKGJ7TV3624DFNCXCK76ZIM5YVKLSQ, seg_pos={0 0}, upload=300780]: Uploading piece...
Jul 9 01:07:56.955856205 (375h57m31.72649754s) [noise=true , address=156.146.43.225:4439 , piece_id=WHIPMS22HXTUQAOUPNXT2DLA6XL2YWXM43KLEC55YFEY2SKBNIBA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12VztoHQmCpGLhyN8wKXqJ9u7BzG9UbbeQinmquUZ9Rb2e2w9Ze, address:156.146.43.225:4439): protocol: storage node overloaded, request limit: 2000
Jul 9 01:07:57.480276305 (375h57m32.25091764s) [upload=300780]: Next returned nil segment
Jul 9 01:07:57.480291304 (375h57m32.250932639s) [upload=300780]: Waiting for error group managing segments...
Jul 9 01:07:57.892917917 (375h57m32.663559252s) [noise=true , address=77.90.31.13:4649 , piece_id=HXJ735B5UBJP4VSLABNV2RDSI6XELDRJ7DPXJXOZHU7QPGY6ON3Q, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.894965904 (375h57m32.66560724s) [noise=true , address=77.90.31.19:4861 , piece_id=RVU6MOFXHFWCAIKRNXKGQT4KADZSLIXXNIG6NDHHNJZ7KEBAXICA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.895386990 (375h57m32.666028325s) [noise=true , address=77.90.31.13:4656 , piece_id=BM62XMC4FBDFBSF7BRQAUE2UWJZU5JUODULKP7CWVPRXFFCGS2CQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.897322493 (375h57m32.667963828s) [noise=true , address=77.90.31.11:4557 , piece_id=IPHAW23VXAVZ4XP3TOMX2RMVO2JDC65HLRRNBSMZTA5JJXY5EABQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.899196428 (375h57m32.669837753s) [noise=true , address=77.90.31.18:4840 , piece_id=SNZW25XL4VJ7UN5ZMGU4ZSOW6FQA5ZEUPQCV5OKRSPQIA3F4XGSA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.901992567 (375h57m32.672633902s) [noise=true , address=77.90.31.14:4677 , piece_id=53ZLF5H2KYVI6HM3GSHVC6JKGJ7TV3624DFNCXCK76ZIM5YVKLSQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.907178669 (375h57m32.677820014s) [noise=true , address=77.90.31.12:4622 , piece_id=F566GJ7G6WUKJE63V7Z25TOJ25LDGSVYHPBYMQCQQLRDTRYBJFYQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.917659831 (375h57m32.688301166s) [noise=true , address=77.90.31.14:4669 , piece_id=KQNMOTC4GWOGYIZY2QK27LKEPN225LAYFJZWCJC6MRCK7QBPRFCQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.928100055 (375h57m32.69874139s) [noise=true , address=77.90.31.17:4801 , piece_id=K2UIXAWUISIA5KCKQCZMBJ3BHB2MS44MVMB7T4T2CCA2PMGJJDDA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:07:57.955811390 (375h57m32.726452725s) [noise=true , address=77.90.31.16:4755 , piece_id=XAPAI7EP2B7XCJI6OBUVPPTPVAHARZVNVGUP4WBMLFPQNBOXUMBQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:08:00.166279605 (375h57m34.93692094s) [noise=true , address=45.91.116.9:4274 , piece_id=XKNACYYGKETXPQTRTATURZ2RRXLIDNUJOG2EKLG46OQUSMN3K33Q, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: storage node overloaded, request limit: 2000
Jul 9 01:08:10.737498024 (375h57m45.508139359s) [noise=true , address=104.36.82.39:4726 , piece_id=5OLQ7AGZ5I7AOKBCSP34VPYIFYB2IVWZJ6WWA4DTDJ5LRNXICUSQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: <nil>
Jul 9 01:08:54.804118607 (375h58m29.574759942s) [noise=true , address=104.36.82.18:4390 , piece_id=G24NDRJ3NCKCWG6OFNUVRXE6HRUDZ7I4P6AKQ446XBKHNIOCD2TQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12AZCJHCR7rF4r62Bg7NcBkpNoLb6r6rEN2a5hZ9KNxJTtjV8jp, address:104.36.82.18:4390): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804258494 (375h58m29.574899819s) [noise=true , address=104.36.82.23:4466 , piece_id=TTUJEA6JYTHG3WPYE5VLFPS6IYRXBUI5IZV3FV5725A6XPMJVZLQ, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12rkNnk9RUhKBJ3EsakqMRb3wYwrhRLhB4ZYbrdfKPDk8erRJpP, address:104.36.82.23:4466): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804300485 (375h58m29.57494184s) [noise=true , address=104.36.82.17:4350 , piece_id=YDPD4LOKKFCGXAGBJGNL7D77APW27OHIYSWYUR7T6XX4ZFP3FHZA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804343667 (375h58m29.574985002s) [noise=true , address=104.36.82.3:4786 , piece_id=24V2D5IBAXBHUDVZ7KJXDNZHVHC2H6IWQ3NAISWJ5PZSPF4TOVWA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:127aTG7tofhLA5799hT6gvZZSkT1vvWWFejx7DS4Zcz8mBHHsuE, address:104.36.82.3:4786): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:08:54.804418601 (375h58m29.575059936s) [noise=true , address=104.36.82.19:4411 , piece_id=CURSEVNLHJHEMVYVLGYHIVTHMYSZSEFLCR5ZDMFSQDREHNBLKSUA, seg_pos={0 0}, upload=300780]: Done uploading piece. err: ecclient: upload failed (node:12nradx4SMYLkuJw3Ad6o9Sq1iNtQR6GDu8qg7tCzZzNcTjVfyd, address:104.36.82.19:4411): protocol: write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection; write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46898->104.36.82.19:4411: use of closed network connection
Jul 9 01:08:54.804449420 (375h58m29.575090755s) [seg_pos={0 0}, upload=300780]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 9 01:08:55.034441116 (375h58m29.805082451s) [upload=300780]: Done uploading segments...
Jul 9 01:08:55.034460744 (375h58m29.805102079s) [upload=300780]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 28.951507663s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X.X................................X........................] duration:
29.095067752s
FAIL (2025-07-09 01:22:30 UTC)
orbiter warning:
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
: read tcp 188.166.225.170:43106->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 9 01:03:42.241129255 (375h53m17.01177059s) [seg_pos={0 0}, upload=300726]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 01:03:42.241207024 (375h53m17.011848359s) [upload=300726]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:03:42.241212213 (375h53m17.011853568s) [upload=300726]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:03:42.555255542 (375h53m17.325896887s) [upload=300726]: Done uploading segments...
Jul 9 01:03:42.555279137 (375h53m17.325920473s) [upload=300726]: Upload finished. err: <nil>
Jul 9 01:04:23.199661990 (375h53m57.970303325s) [upload=300735]: Starting upload
Jul 9 01:04:23.199741541 (375h53m57.970382876s) [upload=300735]: Done starting upload
Jul 9 01:04:23.199829180 (375h53m57.970470515s) [upload=300735]: Uploading segments...
Jul 9 01:04:23.200049922 (375h53m57.970691267s) [upload=300735]: Got next segment. Inline: false
Jul 9 01:04:23.200082845 (375h53m57.97072418s) [upload=300735]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:04:23.200088215 (375h53m57.97072955s) [upload=300735]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:04:23.428281090 (375h53m58.198922415s) [seg_pos={0 0}, upload=300735]: Begin upload segment...
Jul 9 01:04:23.428323781 (375h53m58.198965116s) [seg_pos={0 0}, upload=300735]: Done begin upload segment.
Jul 9 01:04:23.428346174 (375h53m58.198987509s) [noise=true , address=45.91.116.9:4262 , piece_id=ZX4L3FJB4POJ6LAANJH5DAHHJTPSXIEYPTXLWFB44T5CN3EY6EOQ, seg_pos={0 0}, upload=300735]: Uploading piece...
Jul 9 01:04:23.428393515 (375h53m58.19903485s) [noise=true , address=45.91.116.6:4974 , piece_id=VCM4RCJEZAT6NTKEGZR5FLD6CCK5GTS6JDUHM4AX4ODQ7JPQENFQ, seg_pos={0 0}, upload=300735]: Uploading piece...
Jul 9 01:04:23.428732944 (375h53m58.199374279s) [noise=true , address=104.36.82.7:4912 , piece_id=DLFOBK7ACOEMEDQGXWQEIYCLIP7SMSLHXIMZNOQV4MO4HT2G7JHQ, seg_pos={0 0}, upload=300735]: Uploading piece...
Jul 9 01:04:23.428839468 (375h53m58.199480803s) [noise=true , address=104.36.82.15:4258 , piece_id=DAL4ELTDS5ZUVDUS27Q4YUJ7BBHA6AE5IVRGEF45MYVSMLTIIWCA, seg_pos={0 0}, upload=300735]: Uploading piece...
Jul 9 01:04:23.428993011 (375h53m58.199634347s) [noise=true , address=77.90.31.12:4590 , piece_id=BMMJIQ3MRY6GSYGR6WSHWNGXQYWJ2CU5HAV26ZK4AQMHOJB6N6WQ, seg_pos={0 0}, upload=300735]: Uploading piece...
Jul 9 01:04:23.429124573 (375h53m58.199765908s) [noise=true , address=45.91.116.12:4226 , piece_id=QUL5KKTPSSPT764N5U5OYOOBGX7NLGZZLCN2Z6RUCN5RG4XTBJNQ, seg_pos={0 0}, upload=300735]: Uploading piece...
Jul 9 01:04:23.429285602 (375h53m58.199926937s) [noise=true , address=45.91.116.14:4189 , piece_id=LOB5YHYKI6HPC3YVWSJGKYGVSOCHRZIUGEAPUMFUMPSZCCQIOHVA, seg_pos={0 0}, upload=300735]: Uploading piece...
Jul 9 01:04:23.429447592 (375h53m58.200088927s) [noise=true , address=45.91.116.11:4119 , piece_id=LNF43VO6YI2E2IQPDUSQZ2WOKDJR5ZPT76E3V6KBMW6FJ4YBLEKA, seg_pos={0 0}, upload=300735]: Uploading piece...
Jul 9 01:04:23.429666410 (375h53m58.200307735s) [noise=true , address=45.91.116.4:4940 , piece_id=NED7QICPZBWIHY7P7DL3A2TTYBVOSNQJIH4FWSG3B4AA3T3CQQDQ, seg_pos={0 0}, upload=300735]: Uploading piece...
Jul 9 01:04:23.429737987 (375h53m58.200379322s) [noise=true , address=45.91.116.5:4299 , piece_id=F2RRHWD3FLZL7VTEX5LBQ6DE4KCK5FJJDQEHNABF6SJT4OLKRXHQ, seg_pos={0 0}, upload=300735]: Uploading piece...
Jul 9 01:04:23.429946366 (375h53m58.200587701s) [noise=true , address=77.90.31.15:4697 , piece_id=WC2PMITXZUX2SC2TF2DYNWG57RTBRCHSIFFJRZXAPBSYYPN4OSRQ, seg_pos={0 0}, upload=300735]: Uploading piece...
Jul 9 01:04:23.430047119 (375h53m58.200688454s) [noise=true , address=45.91.116.9:4261 , piece_id=SUYDYB2KEJPZYZD7TCQA63Q4MTFMFLVFO6VB53AQETIO4RFIGSPQ, seg_pos={0 0}, upload=300735]: Uploading piece...
Jul 9 01:04:23.430192006 (375h53m58.200833341s) [noise=true , address=45.91.116.8:4048 , piece_id=XNK4HS24U4WUDOQM72WNJZCCZU7CZNCWFV5RSZVOUT2BEOYMHKXA, seg_pos={0 0}, upload=300735]: Uploading piece...
Jul 9 01:04:23.430301075 (375h53m58.20094241s) [noise=true , address=104.36.82.4:4801 , piece_id=RCEMHD2TLCTU3COWEW25OJRPQGIAJH7MVWOQGVZTVABOS7YR65NA, seg_pos={0 0}, upload=300735]: Uploading piece...
Jul 9 01:04:23.430393010 (375h53m58.201034345s) [noise=true , address=45.91.116.6:4975 , piece_id=E3YVVGLAGVYRL4GRM67OGD7VTWGBHVJWPABPZHYTG5VZ7RWQ5RPQ, seg_pos={0 0}, upload=300735]: Uploading piece...
Jul 9 01:04:23.430562204 (375h53m58.201203529s) [noise=true , address=104.36.82.2:4731 , piece_id=XKMPJ5N4VCZZY5CDFHQFDKB6QQAIZWMFNANUBRGXILTLRRDZI5TA, seg_pos={0 0}, upload=300735]: Uploading piece...
Jul 9 01:04:25.216203517 (375h53m59.986844852s) [upload=300735]: Next returned nil segment
Jul 9 01:04:25.216221862 (375h53m59.986863197s) [upload=300735]: Waiting for error group managing segments...
Jul 9 01:04:25.507634604 (375h54m0.278275929s) [noise=true , address=45.91.116.14:4189 , piece_id=LOB5YHYKI6HPC3YVWSJGKYGVSOCHRZIUGEAPUMFUMPSZCCQIOHVA, seg_pos={0 0}, upload=300735]: Done uploading piece. err: <nil>
Jul 9 01:04:25.514928808 (375h54m0.285570143s) [noise=true , address=45.91.116.6:4975 , piece_id=E3YVVGLAGVYRL4GRM67OGD7VTWGBHVJWPABPZHYTG5VZ7RWQ5RPQ, seg_pos={0 0}, upload=300735]: Done uploading piece. err: <nil>
Jul 9 01:04:25.515940684 (375h54m0.286582019s) [noise=true , address=45.91.116.8:4048 , piece_id=XNK4HS24U4WUDOQM72WNJZCCZU7CZNCWFV5RSZVOUT2BEOYMHKXA, seg_pos={0 0}, upload=300735]: Done uploading piece. err: <nil>
Jul 9 01:04:25.519818112 (375h54m0.290459447s) [noise=true , address=45.91.116.5:4299 , piece_id=F2RRHWD3FLZL7VTEX5LBQ6DE4KCK5FJJDQEHNABF6SJT4OLKRXHQ, seg_pos={0 0}, upload=300735]: Done uploading piece. err: <nil>
Jul 9 01:04:25.522328946 (375h54m0.292970271s) [noise=true , address=45.91.116.11:4119 , piece_id=LNF43VO6YI2E2IQPDUSQZ2WOKDJR5ZPT76E3V6KBMW6FJ4YBLEKA, seg_pos={0 0}, upload=300735]: Done uploading piece. err: <nil>
Jul 9 01:04:25.574452014 (375h54m0.345093339s) [noise=true , address=77.90.31.12:4590 , piece_id=BMMJIQ3MRY6GSYGR6WSHWNGXQYWJ2CU5HAV26ZK4AQMHOJB6N6WQ, seg_pos={0 0}, upload=300735]: Done uploading piece. err: <nil>
Jul 9 01:04:25.586797383 (375h54m0.357438718s) [noise=true , address=77.90.31.15:4697 , piece_id=WC2PMITXZUX2SC2TF2DYNWG57RTBRCHSIFFJRZXAPBSYYPN4OSRQ, seg_pos={0 0}, upload=300735]: Done uploading piece. err: <nil>
Jul 9 01:04:25.589635122 (375h54m0.360276457s) [noise=true , address=45.91.116.12:4226 , piece_id=QUL5KKTPSSPT764N5U5OYOOBGX7NLGZZLCN2Z6RUCN5RG4XTBJNQ, seg_pos={0 0}, upload=300735]: Done uploading piece. err: <nil>
Jul 9 01:04:25.589754691 (375h54m0.360396026s) [noise=true , address=45.91.116.4:4940 , piece_id=NED7QICPZBWIHY7P7DL3A2TTYBVOSNQJIH4FWSG3B4AA3T3CQQDQ, seg_pos={0 0}, upload=300735]: Done uploading piece. err: <nil>
Jul 9 01:04:25.605390104 (375h54m0.376031439s) [noise=true , address=45.91.116.9:4262 , piece_id=ZX4L3FJB4POJ6LAANJH5DAHHJTPSXIEYPTXLWFB44T5CN3EY6EOQ, seg_pos={0 0}, upload=300735]: Done uploading piece. err: <nil>
Jul 9 01:04:25.606659412 (375h54m0.377300747s) [noise=true , address=45.91.116.6:4974 , piece_id=VCM4RCJEZAT6NTKEGZR5FLD6CCK5GTS6JDUHM4AX4ODQ7JPQENFQ, seg_pos={0 0}, upload=300735]: Done uploading piece. err: <nil>
Jul 9 01:04:25.656386407 (375h54m0.427027742s) [noise=true , address=45.91.116.9:4261 , piece_id=SUYDYB2KEJPZYZD7TCQA63Q4MTFMFLVFO6VB53AQETIO4RFIGSPQ, seg_pos={0 0}, upload=300735]: Done uploading piece. err: <nil>
Jul 9 01:04:50.159152543 (375h54m24.929793878s) [noise=true , address=104.36.82.15:4258 , piece_id=DAL4ELTDS5ZUVDUS27Q4YUJ7BBHA6AE5IVRGEF45MYVSMLTIIWCA, seg_pos={0 0}, upload=300735]: Done uploading piece. err: <nil>
Jul 9 01:04:50.159180336 (375h54m24.929821671s) [seg_pos={0 0}, upload=300735]: Segment reached optimal threshold of 13 pieces.
Jul 9 01:04:50.159331847 (375h54m24.929973172s) [noise=true , address=104.36.82.4:4801 , piece_id=RCEMHD2TLCTU3COWEW25OJRPQGIAJH7MVWOQGVZTVABOS7YR65NA, seg_pos={0 0}, upload=300735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: expected piece hash; context canceled; EOF
Jul 9 01:04:50.159350792 (375h54m24.929992127s) [noise=true , address=104.36.82.7:4912 , piece_id=DLFOBK7ACOEMEDQGXWQEIYCLIP7SMSLHXIMZNOQV4MO4HT2G7JHQ, seg_pos={0 0}, upload=300735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: expected piece hash; context canceled; EOF
Jul 9 01:04:50.159404315 (375h54m24.93004565s) [noise=true , address=104.36.82.2:4731 , piece_id=XKMPJ5N4VCZZY5CDFHQFDKB6QQAIZWMFNANUBRGXILTLRRDZI5TA, seg_pos={0 0}, upload=300735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: write tcp 188.166.225.170:56532->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:56532->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56532->104.36.82.2:4731: use of closed network connection
Jul 9 01:04:50.159421939 (375h54m24.930063264s) [seg_pos={0 0}, upload=300735]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 01:04:50.159478667 (375h54m24.930120002s) [upload=300735]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:04:50.159482414 (375h54m24.930123749s) [upload=300735]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:04:50.440380963 (375h54m25.211022298s) [upload=300735]: Done uploading segments...
Jul 9 01:04:50.440399008 (375h54m25.211040343s) [upload=300735]: Upload finished. err: <nil>
Duration: 29.095098801s
core
uplink
us-select-1
|