Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXXXXXXX.XX.XXXXXX......X.......XX..........................] duration: 5.908160722s FAIL (2025-08-16 16:19:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.908276113s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................X.................X.....................] duration: 14.665851012s FAIL (2025-08-16 16:19:06 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 14.665876901s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.........................] duration: 6.004129192s FAIL (2025-08-16 16:19:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 6.00415375s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 33 out of 60 is failed (threshold=3) [.X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X................] duration: 11.989352734s FAIL (2025-08-16 16:18:52 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 16 16:05:27.916122149 (190h39m22.34720228s) [upload=198338]: Starting upload Aug 16 16:05:27.916169229 (190h39m22.34724936s) [upload=198338]: Done starting upload Aug 16 16:05:27.916311171 (190h39m22.347391302s) [upload=198338]: Uploading segments... Aug 16 16:05:27.916393368 (190h39m22.347473489s) [upload=198338]: Got next segment. Inline: false Aug 16 16:05:27.916432212 (190h39m22.347512333s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:05:27.916436451 (190h39m22.347516582s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 16:05:34.330059713 (190h39m28.761139904s) [seg_pos={0 0}, upload=198338]: Begin upload segment... Aug 16 16:05:34.330142301 (190h39m28.761222432s) [seg_pos={0 0}, upload=198338]: Done begin upload segment. Aug 16 16:05:34.330182148 (190h39m28.761262289s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330200312 (190h39m28.761280453s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330926893 (190h39m28.762007024s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330942242 (190h39m28.762022373s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331525619 (190h39m28.76260575s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331531049 (190h39m28.76261118s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331942917 (190h39m28.763023058s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331957344 (190h39m28.763037475s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332406254 (190h39m28.763486405s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332521564 (190h39m28.763601705s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332774107 (190h39m28.763854248s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333105160 (190h39m28.764185291s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333429822 (190h39m28.764509963s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333710268 (190h39m28.764790399s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334103742 (190h39m28.765183873s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334579692 (190h39m28.765659833s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334870068 (190h39m28.765950199s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.335273469 (190h39m28.7663536s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:35.811636174 (190h39m30.242716305s) [upload=198338]: Next returned nil segment Aug 16 16:05:35.811656744 (190h39m30.242736875s) [upload=198338]: Waiting for error group managing segments... Aug 16 16:05:36.213433670 (190h39m30.644513801s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.272010409 (190h39m30.70309054s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.328834736 (190h39m30.759914867s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.335734207 (190h39m30.766814328s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.348504793 (190h39m30.779584924s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.375100119 (190h39m30.80618025s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.380335747 (190h39m30.811415878s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.414590841 (190h39m30.845670972s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.437247702 (190h39m30.868327833s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.488147614 (190h39m30.919227745s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.489824552 (190h39m30.920904693s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.495938310 (190h39m30.927018441s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.575349905 (190h39m31.006430036s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.710988976 (190h39m31.142069107s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.711016047 (190h39m31.142096178s) [seg_pos={0 0}, upload=198338]: Segment reached optimal threshold of 14 pieces. Aug 16 16:05:36.711318526 (190h39m31.142398657s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnKJpLn4zQr1Z5BTBVSxbHXRknJh41o24v3GKEduhMRuVb1n9): protocol: expected piece hash; context canceled; EOF Aug 16 16:05:36.711387898 (190h39m31.142468029s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm): protocol: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection Aug 16 16:05:36.711426964 (190h39m31.142507095s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q): protocol: rpc: dial tcp 45.91.116.11:4143: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:05:36.711444947 (190h39m31.142525078s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv): protocol: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection Aug 16 16:05:36.711490996 (190h39m31.142571137s) [seg_pos={0 0}, upload=198338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 16:05:36.711595235 (190h39m31.142675367s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 16:05:36.711601397 (190h39m31.142681538s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:05:38.583410084 (190h39m33.014490215s) [upload=198338]: Done uploading segments... Aug 16 16:05:38.583427637 (190h39m33.014507768s) [upload=198338]: Upload finished. err: <nil> Duration: 11.98938689s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.720373206s FAIL (2025-08-16 16:18:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.720502455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXXXXXXXX.XX.XXXXXX......X.......XX.........................] duration: 5.9106211s FAIL (2025-08-16 16:18:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.910640048s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.834358599s FAIL (2025-08-16 16:18:33 UTC) orbiter warning: context deadline exceeded Duration: 5.834383547s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 33 out of 60 is failed (threshold=3) [..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X...............] duration: 12.99621981s FAIL (2025-08-16 16:18:10 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 16 16:05:27.916122149 (190h39m22.34720228s) [upload=198338]: Starting upload Aug 16 16:05:27.916169229 (190h39m22.34724936s) [upload=198338]: Done starting upload Aug 16 16:05:27.916311171 (190h39m22.347391302s) [upload=198338]: Uploading segments... Aug 16 16:05:27.916393368 (190h39m22.347473489s) [upload=198338]: Got next segment. Inline: false Aug 16 16:05:27.916432212 (190h39m22.347512333s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:05:27.916436451 (190h39m22.347516582s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 16:05:34.330059713 (190h39m28.761139904s) [seg_pos={0 0}, upload=198338]: Begin upload segment... Aug 16 16:05:34.330142301 (190h39m28.761222432s) [seg_pos={0 0}, upload=198338]: Done begin upload segment. Aug 16 16:05:34.330182148 (190h39m28.761262289s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330200312 (190h39m28.761280453s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330926893 (190h39m28.762007024s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330942242 (190h39m28.762022373s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331525619 (190h39m28.76260575s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331531049 (190h39m28.76261118s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331942917 (190h39m28.763023058s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331957344 (190h39m28.763037475s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332406254 (190h39m28.763486405s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332521564 (190h39m28.763601705s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332774107 (190h39m28.763854248s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333105160 (190h39m28.764185291s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333429822 (190h39m28.764509963s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333710268 (190h39m28.764790399s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334103742 (190h39m28.765183873s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334579692 (190h39m28.765659833s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334870068 (190h39m28.765950199s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.335273469 (190h39m28.7663536s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:35.811636174 (190h39m30.242716305s) [upload=198338]: Next returned nil segment Aug 16 16:05:35.811656744 (190h39m30.242736875s) [upload=198338]: Waiting for error group managing segments... Aug 16 16:05:36.213433670 (190h39m30.644513801s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.272010409 (190h39m30.70309054s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.328834736 (190h39m30.759914867s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.335734207 (190h39m30.766814328s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.348504793 (190h39m30.779584924s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.375100119 (190h39m30.80618025s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.380335747 (190h39m30.811415878s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.414590841 (190h39m30.845670972s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.437247702 (190h39m30.868327833s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.488147614 (190h39m30.919227745s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.489824552 (190h39m30.920904693s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.495938310 (190h39m30.927018441s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.575349905 (190h39m31.006430036s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.710988976 (190h39m31.142069107s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.711016047 (190h39m31.142096178s) [seg_pos={0 0}, upload=198338]: Segment reached optimal threshold of 14 pieces. Aug 16 16:05:36.711318526 (190h39m31.142398657s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnKJpLn4zQr1Z5BTBVSxbHXRknJh41o24v3GKEduhMRuVb1n9): protocol: expected piece hash; context canceled; EOF Aug 16 16:05:36.711387898 (190h39m31.142468029s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm): protocol: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection Aug 16 16:05:36.711426964 (190h39m31.142507095s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q): protocol: rpc: dial tcp 45.91.116.11:4143: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:05:36.711444947 (190h39m31.142525078s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv): protocol: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection Aug 16 16:05:36.711490996 (190h39m31.142571137s) [seg_pos={0 0}, upload=198338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 16:05:36.711595235 (190h39m31.142675367s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 16:05:36.711601397 (190h39m31.142681538s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:05:38.583410084 (190h39m33.014490215s) [upload=198338]: Done uploading segments... Aug 16 16:05:38.583427637 (190h39m33.014507768s) [upload=198338]: Upload finished. err: <nil> Duration: 12.996243845s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXX.XX.XXXXXX......X.......XX........................] duration: 6.03414204s FAIL (2025-08-16 16:17:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.034171004s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX........................] duration: 6.176975482s FAIL (2025-08-16 16:17:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 6.177003557s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.................X....................] duration: 6.395217552s FAIL (2025-08-16 16:17:52 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 6.395314417s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.549415028s FAIL (2025-08-16 16:17:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.549438434s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.648209967s FAIL (2025-08-16 16:17:45 UTC) orbiter warning: context deadline exceeded Duration: 5.648231258s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.XXXXXXXXXX.XX.XXXXXX......X.......XX.......................] duration: 5.920137724s FAIL (2025-08-16 16:17:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.920156491s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [.....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......................] duration: 6.214862752s FAIL (2025-08-16 16:17:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 6.214889014s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X..............] duration: 11.79300531s FAIL (2025-08-16 16:17:22 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 16 16:05:27.916122149 (190h39m22.34720228s) [upload=198338]: Starting upload Aug 16 16:05:27.916169229 (190h39m22.34724936s) [upload=198338]: Done starting upload Aug 16 16:05:27.916311171 (190h39m22.347391302s) [upload=198338]: Uploading segments... Aug 16 16:05:27.916393368 (190h39m22.347473489s) [upload=198338]: Got next segment. Inline: false Aug 16 16:05:27.916432212 (190h39m22.347512333s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:05:27.916436451 (190h39m22.347516582s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 16:05:34.330059713 (190h39m28.761139904s) [seg_pos={0 0}, upload=198338]: Begin upload segment... Aug 16 16:05:34.330142301 (190h39m28.761222432s) [seg_pos={0 0}, upload=198338]: Done begin upload segment. Aug 16 16:05:34.330182148 (190h39m28.761262289s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330200312 (190h39m28.761280453s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330926893 (190h39m28.762007024s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330942242 (190h39m28.762022373s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331525619 (190h39m28.76260575s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331531049 (190h39m28.76261118s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331942917 (190h39m28.763023058s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331957344 (190h39m28.763037475s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332406254 (190h39m28.763486405s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332521564 (190h39m28.763601705s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332774107 (190h39m28.763854248s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333105160 (190h39m28.764185291s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333429822 (190h39m28.764509963s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333710268 (190h39m28.764790399s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334103742 (190h39m28.765183873s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334579692 (190h39m28.765659833s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334870068 (190h39m28.765950199s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.335273469 (190h39m28.7663536s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:35.811636174 (190h39m30.242716305s) [upload=198338]: Next returned nil segment Aug 16 16:05:35.811656744 (190h39m30.242736875s) [upload=198338]: Waiting for error group managing segments... Aug 16 16:05:36.213433670 (190h39m30.644513801s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.272010409 (190h39m30.70309054s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.328834736 (190h39m30.759914867s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.335734207 (190h39m30.766814328s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.348504793 (190h39m30.779584924s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.375100119 (190h39m30.80618025s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.380335747 (190h39m30.811415878s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.414590841 (190h39m30.845670972s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.437247702 (190h39m30.868327833s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.488147614 (190h39m30.919227745s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.489824552 (190h39m30.920904693s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.495938310 (190h39m30.927018441s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.575349905 (190h39m31.006430036s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.710988976 (190h39m31.142069107s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.711016047 (190h39m31.142096178s) [seg_pos={0 0}, upload=198338]: Segment reached optimal threshold of 14 pieces. Aug 16 16:05:36.711318526 (190h39m31.142398657s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnKJpLn4zQr1Z5BTBVSxbHXRknJh41o24v3GKEduhMRuVb1n9): protocol: expected piece hash; context canceled; EOF Aug 16 16:05:36.711387898 (190h39m31.142468029s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm): protocol: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection Aug 16 16:05:36.711426964 (190h39m31.142507095s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q): protocol: rpc: dial tcp 45.91.116.11:4143: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:05:36.711444947 (190h39m31.142525078s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv): protocol: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection Aug 16 16:05:36.711490996 (190h39m31.142571137s) [seg_pos={0 0}, upload=198338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 16:05:36.711595235 (190h39m31.142675367s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 16:05:36.711601397 (190h39m31.142681538s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:05:38.583410084 (190h39m33.014490215s) [upload=198338]: Done uploading segments... Aug 16 16:05:38.583427637 (190h39m33.014507768s) [upload=198338]: Upload finished. err: <nil> Duration: 11.793031251s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.182860669s FAIL (2025-08-16 16:16:57 UTC) orbiter warning: context deadline exceeded Duration: 6.182885926s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..XXXXXXXXXX.XX.XXXXXX......X.......XX......................] duration: 6.511331s FAIL (2025-08-16 16:16:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.511363994s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................X.................X...................] duration: 4.426632163s FAIL (2025-08-16 16:16:43 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 4.426658973s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.391673427s FAIL (2025-08-16 16:16:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.391694247s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..XXXXXXXXXX.XX.XXXXXX......X.......XX.....................] duration: 6.091055771s FAIL (2025-08-16 16:16:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.091111226s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [.X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X.............] duration: 11.866208127s FAIL (2025-08-16 16:16:22 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 16 16:05:27.916122149 (190h39m22.34720228s) [upload=198338]: Starting upload Aug 16 16:05:27.916169229 (190h39m22.34724936s) [upload=198338]: Done starting upload Aug 16 16:05:27.916311171 (190h39m22.347391302s) [upload=198338]: Uploading segments... Aug 16 16:05:27.916393368 (190h39m22.347473489s) [upload=198338]: Got next segment. Inline: false Aug 16 16:05:27.916432212 (190h39m22.347512333s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:05:27.916436451 (190h39m22.347516582s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 16:05:34.330059713 (190h39m28.761139904s) [seg_pos={0 0}, upload=198338]: Begin upload segment... Aug 16 16:05:34.330142301 (190h39m28.761222432s) [seg_pos={0 0}, upload=198338]: Done begin upload segment. Aug 16 16:05:34.330182148 (190h39m28.761262289s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330200312 (190h39m28.761280453s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330926893 (190h39m28.762007024s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330942242 (190h39m28.762022373s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331525619 (190h39m28.76260575s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331531049 (190h39m28.76261118s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331942917 (190h39m28.763023058s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331957344 (190h39m28.763037475s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332406254 (190h39m28.763486405s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332521564 (190h39m28.763601705s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332774107 (190h39m28.763854248s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333105160 (190h39m28.764185291s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333429822 (190h39m28.764509963s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333710268 (190h39m28.764790399s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334103742 (190h39m28.765183873s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334579692 (190h39m28.765659833s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334870068 (190h39m28.765950199s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.335273469 (190h39m28.7663536s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:35.811636174 (190h39m30.242716305s) [upload=198338]: Next returned nil segment Aug 16 16:05:35.811656744 (190h39m30.242736875s) [upload=198338]: Waiting for error group managing segments... Aug 16 16:05:36.213433670 (190h39m30.644513801s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.272010409 (190h39m30.70309054s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.328834736 (190h39m30.759914867s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.335734207 (190h39m30.766814328s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.348504793 (190h39m30.779584924s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.375100119 (190h39m30.80618025s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.380335747 (190h39m30.811415878s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.414590841 (190h39m30.845670972s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.437247702 (190h39m30.868327833s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.488147614 (190h39m30.919227745s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.489824552 (190h39m30.920904693s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.495938310 (190h39m30.927018441s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.575349905 (190h39m31.006430036s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.710988976 (190h39m31.142069107s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.711016047 (190h39m31.142096178s) [seg_pos={0 0}, upload=198338]: Segment reached optimal threshold of 14 pieces. Aug 16 16:05:36.711318526 (190h39m31.142398657s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnKJpLn4zQr1Z5BTBVSxbHXRknJh41o24v3GKEduhMRuVb1n9): protocol: expected piece hash; context canceled; EOF Aug 16 16:05:36.711387898 (190h39m31.142468029s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm): protocol: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection Aug 16 16:05:36.711426964 (190h39m31.142507095s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q): protocol: rpc: dial tcp 45.91.116.11:4143: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:05:36.711444947 (190h39m31.142525078s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv): protocol: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection Aug 16 16:05:36.711490996 (190h39m31.142571137s) [seg_pos={0 0}, upload=198338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 16:05:36.711595235 (190h39m31.142675367s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 16:05:36.711601397 (190h39m31.142681538s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:05:38.583410084 (190h39m33.014490215s) [upload=198338]: Done uploading segments... Aug 16 16:05:38.583427637 (190h39m33.014507768s) [upload=198338]: Upload finished. err: <nil> Duration: 11.866234059s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.707840135s FAIL (2025-08-16 16:16:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.707861247s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX......................] duration: 6.266392886s FAIL (2025-08-16 16:16:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 6.266418225s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.973682467s FAIL (2025-08-16 16:16:07 UTC) orbiter warning: context deadline exceeded Duration: 5.973705331s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [..X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X............] duration: 9.687511306s FAIL (2025-08-16 16:15:52 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 16 16:05:27.916122149 (190h39m22.34720228s) [upload=198338]: Starting upload Aug 16 16:05:27.916169229 (190h39m22.34724936s) [upload=198338]: Done starting upload Aug 16 16:05:27.916311171 (190h39m22.347391302s) [upload=198338]: Uploading segments... Aug 16 16:05:27.916393368 (190h39m22.347473489s) [upload=198338]: Got next segment. Inline: false Aug 16 16:05:27.916432212 (190h39m22.347512333s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:05:27.916436451 (190h39m22.347516582s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 16:05:34.330059713 (190h39m28.761139904s) [seg_pos={0 0}, upload=198338]: Begin upload segment... Aug 16 16:05:34.330142301 (190h39m28.761222432s) [seg_pos={0 0}, upload=198338]: Done begin upload segment. Aug 16 16:05:34.330182148 (190h39m28.761262289s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330200312 (190h39m28.761280453s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330926893 (190h39m28.762007024s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330942242 (190h39m28.762022373s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331525619 (190h39m28.76260575s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331531049 (190h39m28.76261118s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331942917 (190h39m28.763023058s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331957344 (190h39m28.763037475s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332406254 (190h39m28.763486405s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332521564 (190h39m28.763601705s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332774107 (190h39m28.763854248s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333105160 (190h39m28.764185291s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333429822 (190h39m28.764509963s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333710268 (190h39m28.764790399s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334103742 (190h39m28.765183873s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334579692 (190h39m28.765659833s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334870068 (190h39m28.765950199s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.335273469 (190h39m28.7663536s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:35.811636174 (190h39m30.242716305s) [upload=198338]: Next returned nil segment Aug 16 16:05:35.811656744 (190h39m30.242736875s) [upload=198338]: Waiting for error group managing segments... Aug 16 16:05:36.213433670 (190h39m30.644513801s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.272010409 (190h39m30.70309054s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.328834736 (190h39m30.759914867s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.335734207 (190h39m30.766814328s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.348504793 (190h39m30.779584924s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.375100119 (190h39m30.80618025s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.380335747 (190h39m30.811415878s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.414590841 (190h39m30.845670972s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.437247702 (190h39m30.868327833s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.488147614 (190h39m30.919227745s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.489824552 (190h39m30.920904693s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.495938310 (190h39m30.927018441s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.575349905 (190h39m31.006430036s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.710988976 (190h39m31.142069107s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.711016047 (190h39m31.142096178s) [seg_pos={0 0}, upload=198338]: Segment reached optimal threshold of 14 pieces. Aug 16 16:05:36.711318526 (190h39m31.142398657s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnKJpLn4zQr1Z5BTBVSxbHXRknJh41o24v3GKEduhMRuVb1n9): protocol: expected piece hash; context canceled; EOF Aug 16 16:05:36.711387898 (190h39m31.142468029s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm): protocol: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection Aug 16 16:05:36.711426964 (190h39m31.142507095s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q): protocol: rpc: dial tcp 45.91.116.11:4143: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:05:36.711444947 (190h39m31.142525078s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv): protocol: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection Aug 16 16:05:36.711490996 (190h39m31.142571137s) [seg_pos={0 0}, upload=198338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 16:05:36.711595235 (190h39m31.142675367s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 16:05:36.711601397 (190h39m31.142681538s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:05:38.583410084 (190h39m33.014490215s) [upload=198338]: Done uploading segments... Aug 16 16:05:38.583427637 (190h39m33.014507768s) [upload=198338]: Upload finished. err: <nil> Duration: 9.687538848s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X..XXXXXXXXXX.XX.XXXXXX......X.......XX....................] duration: 5.838363054s FAIL (2025-08-16 16:15:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.838405846s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [X......................X.................X..................] duration: 6.641746446s FAIL (2025-08-16 16:15:33 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 6.641768327s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....................] duration: 6.101115729s FAIL (2025-08-16 16:15:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 6.101142631s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.728790474s FAIL (2025-08-16 16:15:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.728813248s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.163815795s FAIL (2025-08-16 16:15:20 UTC) orbiter warning: context deadline exceeded Duration: 6.163838709s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....................] duration: 5.976743818s FAIL (2025-08-16 16:15:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 5.976767644s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.X......................X.................X.................] duration: 6.848097287s FAIL (2025-08-16 16:15:03 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 6.848131301s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [...X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X...........] duration: 12.770449527s FAIL (2025-08-16 16:14:51 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 16 16:05:27.916122149 (190h39m22.34720228s) [upload=198338]: Starting upload Aug 16 16:05:27.916169229 (190h39m22.34724936s) [upload=198338]: Done starting upload Aug 16 16:05:27.916311171 (190h39m22.347391302s) [upload=198338]: Uploading segments... Aug 16 16:05:27.916393368 (190h39m22.347473489s) [upload=198338]: Got next segment. Inline: false Aug 16 16:05:27.916432212 (190h39m22.347512333s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:05:27.916436451 (190h39m22.347516582s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 16:05:34.330059713 (190h39m28.761139904s) [seg_pos={0 0}, upload=198338]: Begin upload segment... Aug 16 16:05:34.330142301 (190h39m28.761222432s) [seg_pos={0 0}, upload=198338]: Done begin upload segment. Aug 16 16:05:34.330182148 (190h39m28.761262289s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330200312 (190h39m28.761280453s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330926893 (190h39m28.762007024s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330942242 (190h39m28.762022373s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331525619 (190h39m28.76260575s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331531049 (190h39m28.76261118s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331942917 (190h39m28.763023058s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331957344 (190h39m28.763037475s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332406254 (190h39m28.763486405s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332521564 (190h39m28.763601705s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332774107 (190h39m28.763854248s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333105160 (190h39m28.764185291s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333429822 (190h39m28.764509963s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333710268 (190h39m28.764790399s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334103742 (190h39m28.765183873s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334579692 (190h39m28.765659833s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334870068 (190h39m28.765950199s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.335273469 (190h39m28.7663536s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:35.811636174 (190h39m30.242716305s) [upload=198338]: Next returned nil segment Aug 16 16:05:35.811656744 (190h39m30.242736875s) [upload=198338]: Waiting for error group managing segments... Aug 16 16:05:36.213433670 (190h39m30.644513801s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.272010409 (190h39m30.70309054s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.328834736 (190h39m30.759914867s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.335734207 (190h39m30.766814328s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.348504793 (190h39m30.779584924s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.375100119 (190h39m30.80618025s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.380335747 (190h39m30.811415878s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.414590841 (190h39m30.845670972s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.437247702 (190h39m30.868327833s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.488147614 (190h39m30.919227745s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.489824552 (190h39m30.920904693s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.495938310 (190h39m30.927018441s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.575349905 (190h39m31.006430036s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.710988976 (190h39m31.142069107s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.711016047 (190h39m31.142096178s) [seg_pos={0 0}, upload=198338]: Segment reached optimal threshold of 14 pieces. Aug 16 16:05:36.711318526 (190h39m31.142398657s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnKJpLn4zQr1Z5BTBVSxbHXRknJh41o24v3GKEduhMRuVb1n9): protocol: expected piece hash; context canceled; EOF Aug 16 16:05:36.711387898 (190h39m31.142468029s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm): protocol: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection Aug 16 16:05:36.711426964 (190h39m31.142507095s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q): protocol: rpc: dial tcp 45.91.116.11:4143: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:05:36.711444947 (190h39m31.142525078s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv): protocol: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection Aug 16 16:05:36.711490996 (190h39m31.142571137s) [seg_pos={0 0}, upload=198338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 16:05:36.711595235 (190h39m31.142675367s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 16:05:36.711601397 (190h39m31.142681538s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:05:38.583410084 (190h39m33.014490215s) [upload=198338]: Done uploading segments... Aug 16 16:05:38.583427637 (190h39m33.014507768s) [upload=198338]: Upload finished. err: <nil> Duration: 12.770473922s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...................] duration: 5.938963529s FAIL (2025-08-16 16:14:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 5.938994639s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X..XXXXXXXXXX.XX.XXXXXX......X.......XX...................] duration: 5.992500176s FAIL (2025-08-16 16:14:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.992527598s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.101633569s FAIL (2025-08-16 16:14:48 UTC) orbiter warning: context deadline exceeded Duration: 6.101648809s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.555695135s FAIL (2025-08-16 16:14:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.555711195s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...X..XXXXXXXXXX.XX.XXXXXX......X.......XX..................] duration: 6.000954169s FAIL (2025-08-16 16:14:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.000980319s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.029528763s FAIL (2025-08-16 16:14:21 UTC) orbiter warning: context deadline exceeded Duration: 6.029550044s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..X......................X.................X................] duration: 4.397130123s FAIL (2025-08-16 16:14:04 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 4.397157486s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.36791035s FAIL (2025-08-16 16:14:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.367928324s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [....X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X..........] duration: 11.336524136s FAIL (2025-08-16 16:14:02 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 16 16:05:27.916122149 (190h39m22.34720228s) [upload=198338]: Starting upload Aug 16 16:05:27.916169229 (190h39m22.34724936s) [upload=198338]: Done starting upload Aug 16 16:05:27.916311171 (190h39m22.347391302s) [upload=198338]: Uploading segments... Aug 16 16:05:27.916393368 (190h39m22.347473489s) [upload=198338]: Got next segment. Inline: false Aug 16 16:05:27.916432212 (190h39m22.347512333s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:05:27.916436451 (190h39m22.347516582s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 16:05:34.330059713 (190h39m28.761139904s) [seg_pos={0 0}, upload=198338]: Begin upload segment... Aug 16 16:05:34.330142301 (190h39m28.761222432s) [seg_pos={0 0}, upload=198338]: Done begin upload segment. Aug 16 16:05:34.330182148 (190h39m28.761262289s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330200312 (190h39m28.761280453s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330926893 (190h39m28.762007024s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330942242 (190h39m28.762022373s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331525619 (190h39m28.76260575s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331531049 (190h39m28.76261118s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331942917 (190h39m28.763023058s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331957344 (190h39m28.763037475s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332406254 (190h39m28.763486405s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332521564 (190h39m28.763601705s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332774107 (190h39m28.763854248s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333105160 (190h39m28.764185291s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333429822 (190h39m28.764509963s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333710268 (190h39m28.764790399s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334103742 (190h39m28.765183873s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334579692 (190h39m28.765659833s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334870068 (190h39m28.765950199s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.335273469 (190h39m28.7663536s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:35.811636174 (190h39m30.242716305s) [upload=198338]: Next returned nil segment Aug 16 16:05:35.811656744 (190h39m30.242736875s) [upload=198338]: Waiting for error group managing segments... Aug 16 16:05:36.213433670 (190h39m30.644513801s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.272010409 (190h39m30.70309054s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.328834736 (190h39m30.759914867s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.335734207 (190h39m30.766814328s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.348504793 (190h39m30.779584924s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.375100119 (190h39m30.80618025s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.380335747 (190h39m30.811415878s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.414590841 (190h39m30.845670972s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.437247702 (190h39m30.868327833s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.488147614 (190h39m30.919227745s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.489824552 (190h39m30.920904693s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.495938310 (190h39m30.927018441s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.575349905 (190h39m31.006430036s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.710988976 (190h39m31.142069107s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.711016047 (190h39m31.142096178s) [seg_pos={0 0}, upload=198338]: Segment reached optimal threshold of 14 pieces. Aug 16 16:05:36.711318526 (190h39m31.142398657s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnKJpLn4zQr1Z5BTBVSxbHXRknJh41o24v3GKEduhMRuVb1n9): protocol: expected piece hash; context canceled; EOF Aug 16 16:05:36.711387898 (190h39m31.142468029s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm): protocol: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection Aug 16 16:05:36.711426964 (190h39m31.142507095s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q): protocol: rpc: dial tcp 45.91.116.11:4143: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:05:36.711444947 (190h39m31.142525078s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv): protocol: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection Aug 16 16:05:36.711490996 (190h39m31.142571137s) [seg_pos={0 0}, upload=198338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 16:05:36.711595235 (190h39m31.142675367s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 16:05:36.711601397 (190h39m31.142681538s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:05:38.583410084 (190h39m33.014490215s) [upload=198338]: Done uploading segments... Aug 16 16:05:38.583427637 (190h39m33.014507768s) [upload=198338]: Upload finished. err: <nil> Duration: 11.336555086s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..................] duration: 5.886162321s FAIL (2025-08-16 16:13:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 5.886190727s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.534401412s FAIL (2025-08-16 16:13:45 UTC) orbiter warning: context deadline exceeded Duration: 6.534426569s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....X..XXXXXXXXXX.XX.XXXXXX......X.......XX.................] duration: 6.294081345s FAIL (2025-08-16 16:13:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.294108609s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...X......................X.................X...............] duration: 7.974098863s FAIL (2025-08-16 16:13:29 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 7.974150432s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.................] duration: 6.009406804s FAIL (2025-08-16 16:13:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 6.009443324s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.727614121s FAIL (2025-08-16 16:13:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.72764023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [.....X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X.........] duration: 11.50522804s FAIL (2025-08-16 16:13:14 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 16 16:05:27.916122149 (190h39m22.34720228s) [upload=198338]: Starting upload Aug 16 16:05:27.916169229 (190h39m22.34724936s) [upload=198338]: Done starting upload Aug 16 16:05:27.916311171 (190h39m22.347391302s) [upload=198338]: Uploading segments... Aug 16 16:05:27.916393368 (190h39m22.347473489s) [upload=198338]: Got next segment. Inline: false Aug 16 16:05:27.916432212 (190h39m22.347512333s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:05:27.916436451 (190h39m22.347516582s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 16:05:34.330059713 (190h39m28.761139904s) [seg_pos={0 0}, upload=198338]: Begin upload segment... Aug 16 16:05:34.330142301 (190h39m28.761222432s) [seg_pos={0 0}, upload=198338]: Done begin upload segment. Aug 16 16:05:34.330182148 (190h39m28.761262289s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330200312 (190h39m28.761280453s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330926893 (190h39m28.762007024s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330942242 (190h39m28.762022373s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331525619 (190h39m28.76260575s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331531049 (190h39m28.76261118s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331942917 (190h39m28.763023058s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331957344 (190h39m28.763037475s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332406254 (190h39m28.763486405s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332521564 (190h39m28.763601705s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332774107 (190h39m28.763854248s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333105160 (190h39m28.764185291s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333429822 (190h39m28.764509963s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333710268 (190h39m28.764790399s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334103742 (190h39m28.765183873s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334579692 (190h39m28.765659833s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334870068 (190h39m28.765950199s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.335273469 (190h39m28.7663536s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:35.811636174 (190h39m30.242716305s) [upload=198338]: Next returned nil segment Aug 16 16:05:35.811656744 (190h39m30.242736875s) [upload=198338]: Waiting for error group managing segments... Aug 16 16:05:36.213433670 (190h39m30.644513801s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.272010409 (190h39m30.70309054s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.328834736 (190h39m30.759914867s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.335734207 (190h39m30.766814328s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.348504793 (190h39m30.779584924s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.375100119 (190h39m30.80618025s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.380335747 (190h39m30.811415878s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.414590841 (190h39m30.845670972s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.437247702 (190h39m30.868327833s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.488147614 (190h39m30.919227745s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.489824552 (190h39m30.920904693s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.495938310 (190h39m30.927018441s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.575349905 (190h39m31.006430036s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.710988976 (190h39m31.142069107s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.711016047 (190h39m31.142096178s) [seg_pos={0 0}, upload=198338]: Segment reached optimal threshold of 14 pieces. Aug 16 16:05:36.711318526 (190h39m31.142398657s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnKJpLn4zQr1Z5BTBVSxbHXRknJh41o24v3GKEduhMRuVb1n9): protocol: expected piece hash; context canceled; EOF Aug 16 16:05:36.711387898 (190h39m31.142468029s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm): protocol: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection Aug 16 16:05:36.711426964 (190h39m31.142507095s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q): protocol: rpc: dial tcp 45.91.116.11:4143: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:05:36.711444947 (190h39m31.142525078s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv): protocol: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection Aug 16 16:05:36.711490996 (190h39m31.142571137s) [seg_pos={0 0}, upload=198338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 16:05:36.711595235 (190h39m31.142675367s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 16:05:36.711601397 (190h39m31.142681538s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:05:38.583410084 (190h39m33.014490215s) [upload=198338]: Done uploading segments... Aug 16 16:05:38.583427637 (190h39m33.014507768s) [upload=198338]: Upload finished. err: <nil> Duration: 11.505254773s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)