Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........X.........................................X.........] duration: 1.970877689s FAIL (2025-08-08 00:58:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== stItem_ObjectCommit Aug 8 00:42:17.713078039 (15h2m17.302423046s) [upload=15574]: Done uploading segments... Aug 8 00:42:17.713100702 (15h2m17.30244574s) [upload=15574]: Upload finished. err: <nil> Aug 8 00:42:36.033866449 (15h2m35.623211436s) [upload=15582]: Starting upload Aug 8 00:42:36.033918057 (15h2m35.623263044s) [upload=15582]: Done starting upload Aug 8 00:42:36.033968163 (15h2m35.623313141s) [upload=15582]: Uploading segments... Aug 8 00:42:36.033998922 (15h2m35.623343909s) [upload=15582]: Got next segment. Inline: true Aug 8 00:42:36.034002399 (15h2m35.623347386s) [upload=15582]: Waiting for error group managing segments... Aug 8 00:42:36.034039881 (15h2m35.623384868s) [upload=15582]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:42:36.034043698 (15h2m35.623388685s) [upload=15582]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:42:36.034046063 (15h2m35.62339105s) [upload=15582]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:42:36.326408369 (15h2m35.915753356s) [upload=15582]: Done uploading segments... Aug 8 00:42:36.326427806 (15h2m35.915772823s) [upload=15582]: Upload finished. err: <nil> Aug 8 00:43:50.785976974 (15h3m50.375321961s) [upload=15600]: Starting upload Aug 8 00:43:50.786021608 (15h3m50.375366585s) [upload=15600]: Done starting upload Aug 8 00:43:50.786068328 (15h3m50.375413315s) [upload=15600]: Uploading segments... Aug 8 00:43:50.786096292 (15h3m50.375441269s) [upload=15600]: Got next segment. Inline: true Aug 8 00:43:50.786099928 (15h3m50.375444915s) [upload=15600]: Waiting for error group managing segments... Aug 8 00:43:50.786133022 (15h3m50.375478009s) [upload=15600]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:43:50.786136840 (15h3m50.375481817s) [upload=15600]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:43:50.786139374 (15h3m50.375484351s) [upload=15600]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:43:51.071276196 (15h3m50.660621183s) [upload=15600]: Done uploading segments... Aug 8 00:43:51.071294140 (15h3m50.660639127s) [upload=15600]: Upload finished. err: <nil> Aug 8 00:44:38.932060864 (15h4m38.521405851s) [upload=15614]: Starting upload Aug 8 00:44:38.932105570 (15h4m38.521450547s) [upload=15614]: Done starting upload Aug 8 00:44:38.932150475 (15h4m38.521495462s) [upload=15614]: Uploading segments... Aug 8 00:44:38.932180272 (15h4m38.521525249s) [upload=15614]: Got next segment. Inline: true Aug 8 00:44:38.932183829 (15h4m38.521528816s) [upload=15614]: Waiting for error group managing segments... Aug 8 00:44:38.932214318 (15h4m38.521559305s) [upload=15614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:44:38.932218205 (15h4m38.521563192s) [upload=15614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:44:38.932220419 (15h4m38.521565396s) [upload=15614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:44:39.257010779 (15h4m38.846355757s) [upload=15614]: Done uploading segments... Aug 8 00:44:39.257027903 (15h4m38.84637289s) [upload=15614]: Upload finished. err: <nil> Aug 8 00:44:55.671929119 (15h4m55.261274106s) [upload=15619]: Starting upload Aug 8 00:44:55.671974105 (15h4m55.261319092s) [upload=15619]: Done starting upload Aug 8 00:44:55.672033319 (15h4m55.261378296s) [upload=15619]: Uploading segments... Aug 8 00:44:55.672073296 (15h4m55.261418273s) [upload=15619]: Got next segment. Inline: true Aug 8 00:44:55.672077514 (15h4m55.261422501s) [upload=15619]: Waiting for error group managing segments... Aug 8 00:44:55.672111548 (15h4m55.261456535s) [upload=15619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:44:55.672115085 (15h4m55.261460072s) [upload=15619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:44:55.672117520 (15h4m55.261462507s) [upload=15619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:44:55.946154002 (15h4m55.535498989s) [upload=15619]: Done uploading segments... Aug 8 00:44:55.946180583 (15h4m55.53552557s) [upload=15619]: Upload finished. err: <nil> Aug 8 00:45:51.335687032 (15h5m50.925032019s) [upload=15634]: Starting upload Aug 8 00:45:51.335741947 (15h5m50.925086934s) [upload=15634]: Done starting upload Aug 8 00:45:51.335838182 (15h5m50.925183169s) [upload=15634]: Uploading segments... Aug 8 00:45:51.335870213 (15h5m50.9252152s) [upload=15634]: Got next segment. Inline: true Aug 8 00:45:51.335873930 (15h5m50.925218917s) [upload=15634]: Waiting for error group managing segments... Aug 8 00:45:51.335917033 (15h5m50.92526202s) [upload=15634]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:45:51.335921060 (15h5m50.925266047s) [upload=15634]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:45:51.335923345 (15h5m50.925268332s) [upload=15634]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:45:51.622624488 (15h5m51.211969475s) [upload=15634]: Done uploading segments... Aug 8 00:45:51.622650428 (15h5m51.211995415s) [upload=15634]: Upload finished. err: <nil> Aug 8 00:46:56.158030766 (15h6m55.747375753s) [upload=15655]: Starting upload Aug 8 00:46:56.158075261 (15h6m55.747420248s) [upload=15655]: Done starting upload Aug 8 00:46:56.158127792 (15h6m55.747472779s) [upload=15655]: Uploading segments... Aug 8 00:46:56.158159252 (15h6m55.747504229s) [upload=15655]: Got next segment. Inline: true Aug 8 00:46:56.158162568 (15h6m55.747507555s) [upload=15655]: Waiting for error group managing segments... Aug 8 00:46:56.158193087 (15h6m55.747538074s) [upload=15655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:46:56.158196653 (15h6m55.74754164s) [upload=15655]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:46:56.158199098 (15h6m55.747544075s) [upload=15655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:46:56.421913161 (15h6m56.011258168s) [upload=15655]: Done uploading segments... Aug 8 00:46:56.421936716 (15h6m56.011281703s) [upload=15655]: Upload finished. err: <nil> Aug 8 00:48:03.458104986 (15h8m3.047449973s) [upload=15670]: Starting upload Aug 8 00:48:03.458166774 (15h8m3.047511761s) [upload=15670]: Done starting upload Aug 8 00:48:03.458226910 (15h8m3.047571887s) [upload=15670]: Uploading segments... Aug 8 00:48:03.458331268 (15h8m3.047676255s) [upload=15670]: Got next segment. Inline: true Aug 8 00:48:03.458336880 (15h8m3.047681857s) [upload=15670]: Waiting for error group managing segments... Aug 8 00:48:03.458383799 (15h8m3.047728786s) [upload=15670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:48:03.458388839 (15h8m3.047733827s) [upload=15670]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:48:03.458391154 (15h8m3.047736141s) [upload=15670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:48:03.756480234 (15h8m3.345825221s) [upload=15670]: Done uploading segments... Aug 8 00:48:03.756500813 (15h8m3.3458458s) [upload=15670]: Upload finished. err: <nil> Aug 8 00:49:01.779484346 (15h9m1.368829333s) [upload=15683]: Starting upload Aug 8 00:49:01.779541455 (15h9m1.368886442s) [upload=15683]: Done starting upload Aug 8 00:49:01.779591250 (15h9m1.368936238s) [upload=15683]: Uploading segments... Aug 8 00:49:01.779632399 (15h9m1.368977386s) [upload=15683]: Got next segment. Inline: true Aug 8 00:49:01.779637188 (15h9m1.368982185s) [upload=15683]: Waiting for error group managing segments... Aug 8 00:49:01.779673187 (15h9m1.369018174s) [upload=15683]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:49:01.779677375 (15h9m1.369022362s) [upload=15683]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:49:01.779680040 (15h9m1.369025027s) [upload=15683]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:49:02.060683371 (15h9m1.650028378s) [upload=15683]: Done uploading segments... Aug 8 00:49:02.060710513 (15h9m1.6500555s) [upload=15683]: Upload finished. err: <nil> Aug 8 00:50:06.726842474 (15h10m6.316187461s) [upload=15705]: Starting upload Aug 8 00:50:06.726885055 (15h10m6.316230032s) [upload=15705]: Done starting upload Aug 8 00:50:06.726932025 (15h10m6.316277012s) [upload=15705]: Uploading segments... Aug 8 00:50:06.726962704 (15h10m6.316307691s) [upload=15705]: Got next segment. Inline: true Aug 8 00:50:06.726966831 (15h10m6.316311808s) [upload=15705]: Waiting for error group managing segments... Aug 8 00:50:06.726999844 (15h10m6.316344831s) [upload=15705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:50:06.727003823 (15h10m6.31634881s) [upload=15705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:50:06.727006187 (15h10m6.316351164s) [upload=15705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:50:07.002622120 (15h10m6.591967127s) [upload=15705]: Done uploading segments... Aug 8 00:50:07.002639403 (15h10m6.59198438s) [upload=15705]: Upload finished. err: <nil> Aug 8 00:51:18.800365702 (15h11m18.389710689s) [upload=15723]: Starting upload Aug 8 00:51:18.800416448 (15h11m18.389761435s) [upload=15723]: Done starting upload Aug 8 00:51:18.800490300 (15h11m18.389835277s) [upload=15723]: Uploading segments... Aug 8 00:51:18.800624426 (15h11m18.389969413s) [upload=15723]: Got next segment. Inline: true Aug 8 00:51:18.800629807 (15h11m18.389974794s) [upload=15723]: Waiting for error group managing segments... Aug 8 00:51:18.800677268 (15h11m18.390022255s) [upload=15723]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:51:18.800682698 (15h11m18.390027685s) [upload=15723]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:51:18.800685353 (15h11m18.39003034s) [upload=15723]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:51:19.015137684 (15h11m18.604482671s) [upload=15723]: Done uploading segments... Aug 8 00:51:19.015155639 (15h11m18.604500616s) [upload=15723]: Upload finished. err: metaclient: Too Many Requests Duration: 1.970918528s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.......X...X............X....X.........................] duration: 5.678558987s FAIL (2025-08-08 00:58:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837172821 (14h59m54.426517798s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837385949 (14h59m54.426730936s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837507802 (14h59m54.426852779s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837640145 (14h59m54.426985122s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837819459 (14h59m54.427164446s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837962502 (14h59m54.427307479s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838112790 (14h59m54.427457767s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838238691 (14h59m54.427583678s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838393447 (14h59m54.427738434s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838490402 (14h59m54.427835389s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838660288 (14h59m54.428005275s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838681548 (14h59m54.428026525s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838866312 (14h59m54.428211299s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838893975 (14h59m54.428238962s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.839056986 (14h59m54.428401974s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:55.935384249 (14h59m55.524729246s) [upload=15536]: Next returned nil segment Aug 8 00:39:55.935402815 (14h59m55.524747802s) [upload=15536]: Waiting for error group managing segments... Aug 8 00:39:56.226788010 (14h59m55.816132997s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.265710491 (14h59m55.855055478s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.326363980 (14h59m55.915708967s) [noise=true , address=64.49.13.169:4709 , piece_id=PIPH24KH4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773039652 (14h59m56.362384679s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773213735 (14h59m56.362558732s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.782851711 (14h59m56.372196698s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854376586 (14h59m56.443721573s) [noise=true , address=104.36.82.46:4484 , piece_id=BMHSO2DIXVRRHUZRHWUQ24TXMMTRMVX34SVDUUUMRMKPSUQG3APQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854703231 (14h59m56.444048218s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.914818370 (14h59m56.504163357s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.915843802 (14h59m56.505188849s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.942005621 (14h59m56.531350608s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.981103286 (14h59m56.570448273s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.095564053 (14h59m56.68490904s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216182613 (14h59m56.8055276s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216206869 (14h59m56.805551856s) [seg_pos={0 0}, upload=15536]: Segment reached optimal threshold of 14 pieces. Aug 8 00:39:57.216406632 (14h59m56.805751619s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bN25eEEjxCWKJSr8qj7moDJxArgfsjzPTdBcLdfvoeyxiqPpc): protocol: expected piece hash; context canceled; EOF Aug 8 00:39:57.216460465 (14h59m56.805805452s) [noise=true , address=104.36.82.5:4869 , piece_id=JBCDCG676BBV6U2GAVWDQIWYXKKQGW2DKK6MDY74LNPGP4K3CTQA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection Aug 8 00:39:57.216644467 (14h59m56.805989454s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD): protocol: read tcp 188.166.225.170:50728->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216665567 (14h59m56.806010554s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: read tcp 188.166.225.170:45052->104.36.82.61:4383: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216679534 (14h59m56.806024521s) [seg_pos={0 0}, upload=15536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:39:57.216739699 (14h59m56.806084686s) [upload=15536]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:39:57.216743286 (14h59m56.806088273s) [upload=15536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:39:57.516798834 (14h59m57.106143821s) [upload=15536]: Done uploading segments... Aug 8 00:39:57.516817129 (14h59m57.106162116s) [upload=15536]: Upload finished. err: <nil> Aug 8 00:40:09.579830255 (15h0m9.169175242s) [upload=15541]: Starting upload Aug 8 00:40:09.579877025 (15h0m9.169222012s) [upload=15541]: Done starting upload Aug 8 00:40:09.579941739 (15h0m9.169286726s) [upload=15541]: Uploading segments... Aug 8 00:40:09.580164405 (15h0m9.169509432s) [upload=15541]: Got next segment. Inline: false Aug 8 00:40:09.580219460 (15h0m9.169564477s) [upload=15541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:40:09.580251472 (15h0m9.169596489s) [upload=15541]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:40:09.797368422 (15h0m9.386713409s) [upload=15541]: Next segment err: metaclient: Too Many Requests Aug 8 00:40:09.797398149 (15h0m9.386743136s) [upload=15541]: Done uploading segments... Aug 8 00:40:09.797473614 (15h0m9.386818601s) [upload=15541]: Upload finished. err: metaclient: Too Many Requests Duration: 5.678590438s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........X.........................................X........] duration: 2.285239507s FAIL (2025-08-08 00:57:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== stItem_ObjectCommit Aug 8 00:42:17.713078039 (15h2m17.302423046s) [upload=15574]: Done uploading segments... Aug 8 00:42:17.713100702 (15h2m17.30244574s) [upload=15574]: Upload finished. err: <nil> Aug 8 00:42:36.033866449 (15h2m35.623211436s) [upload=15582]: Starting upload Aug 8 00:42:36.033918057 (15h2m35.623263044s) [upload=15582]: Done starting upload Aug 8 00:42:36.033968163 (15h2m35.623313141s) [upload=15582]: Uploading segments... Aug 8 00:42:36.033998922 (15h2m35.623343909s) [upload=15582]: Got next segment. Inline: true Aug 8 00:42:36.034002399 (15h2m35.623347386s) [upload=15582]: Waiting for error group managing segments... Aug 8 00:42:36.034039881 (15h2m35.623384868s) [upload=15582]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:42:36.034043698 (15h2m35.623388685s) [upload=15582]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:42:36.034046063 (15h2m35.62339105s) [upload=15582]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:42:36.326408369 (15h2m35.915753356s) [upload=15582]: Done uploading segments... Aug 8 00:42:36.326427806 (15h2m35.915772823s) [upload=15582]: Upload finished. err: <nil> Aug 8 00:43:50.785976974 (15h3m50.375321961s) [upload=15600]: Starting upload Aug 8 00:43:50.786021608 (15h3m50.375366585s) [upload=15600]: Done starting upload Aug 8 00:43:50.786068328 (15h3m50.375413315s) [upload=15600]: Uploading segments... Aug 8 00:43:50.786096292 (15h3m50.375441269s) [upload=15600]: Got next segment. Inline: true Aug 8 00:43:50.786099928 (15h3m50.375444915s) [upload=15600]: Waiting for error group managing segments... Aug 8 00:43:50.786133022 (15h3m50.375478009s) [upload=15600]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:43:50.786136840 (15h3m50.375481817s) [upload=15600]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:43:50.786139374 (15h3m50.375484351s) [upload=15600]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:43:51.071276196 (15h3m50.660621183s) [upload=15600]: Done uploading segments... Aug 8 00:43:51.071294140 (15h3m50.660639127s) [upload=15600]: Upload finished. err: <nil> Aug 8 00:44:38.932060864 (15h4m38.521405851s) [upload=15614]: Starting upload Aug 8 00:44:38.932105570 (15h4m38.521450547s) [upload=15614]: Done starting upload Aug 8 00:44:38.932150475 (15h4m38.521495462s) [upload=15614]: Uploading segments... Aug 8 00:44:38.932180272 (15h4m38.521525249s) [upload=15614]: Got next segment. Inline: true Aug 8 00:44:38.932183829 (15h4m38.521528816s) [upload=15614]: Waiting for error group managing segments... Aug 8 00:44:38.932214318 (15h4m38.521559305s) [upload=15614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:44:38.932218205 (15h4m38.521563192s) [upload=15614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:44:38.932220419 (15h4m38.521565396s) [upload=15614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:44:39.257010779 (15h4m38.846355757s) [upload=15614]: Done uploading segments... Aug 8 00:44:39.257027903 (15h4m38.84637289s) [upload=15614]: Upload finished. err: <nil> Aug 8 00:44:55.671929119 (15h4m55.261274106s) [upload=15619]: Starting upload Aug 8 00:44:55.671974105 (15h4m55.261319092s) [upload=15619]: Done starting upload Aug 8 00:44:55.672033319 (15h4m55.261378296s) [upload=15619]: Uploading segments... Aug 8 00:44:55.672073296 (15h4m55.261418273s) [upload=15619]: Got next segment. Inline: true Aug 8 00:44:55.672077514 (15h4m55.261422501s) [upload=15619]: Waiting for error group managing segments... Aug 8 00:44:55.672111548 (15h4m55.261456535s) [upload=15619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:44:55.672115085 (15h4m55.261460072s) [upload=15619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:44:55.672117520 (15h4m55.261462507s) [upload=15619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:44:55.946154002 (15h4m55.535498989s) [upload=15619]: Done uploading segments... Aug 8 00:44:55.946180583 (15h4m55.53552557s) [upload=15619]: Upload finished. err: <nil> Aug 8 00:45:51.335687032 (15h5m50.925032019s) [upload=15634]: Starting upload Aug 8 00:45:51.335741947 (15h5m50.925086934s) [upload=15634]: Done starting upload Aug 8 00:45:51.335838182 (15h5m50.925183169s) [upload=15634]: Uploading segments... Aug 8 00:45:51.335870213 (15h5m50.9252152s) [upload=15634]: Got next segment. Inline: true Aug 8 00:45:51.335873930 (15h5m50.925218917s) [upload=15634]: Waiting for error group managing segments... Aug 8 00:45:51.335917033 (15h5m50.92526202s) [upload=15634]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:45:51.335921060 (15h5m50.925266047s) [upload=15634]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:45:51.335923345 (15h5m50.925268332s) [upload=15634]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:45:51.622624488 (15h5m51.211969475s) [upload=15634]: Done uploading segments... Aug 8 00:45:51.622650428 (15h5m51.211995415s) [upload=15634]: Upload finished. err: <nil> Aug 8 00:46:56.158030766 (15h6m55.747375753s) [upload=15655]: Starting upload Aug 8 00:46:56.158075261 (15h6m55.747420248s) [upload=15655]: Done starting upload Aug 8 00:46:56.158127792 (15h6m55.747472779s) [upload=15655]: Uploading segments... Aug 8 00:46:56.158159252 (15h6m55.747504229s) [upload=15655]: Got next segment. Inline: true Aug 8 00:46:56.158162568 (15h6m55.747507555s) [upload=15655]: Waiting for error group managing segments... Aug 8 00:46:56.158193087 (15h6m55.747538074s) [upload=15655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:46:56.158196653 (15h6m55.74754164s) [upload=15655]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:46:56.158199098 (15h6m55.747544075s) [upload=15655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:46:56.421913161 (15h6m56.011258168s) [upload=15655]: Done uploading segments... Aug 8 00:46:56.421936716 (15h6m56.011281703s) [upload=15655]: Upload finished. err: <nil> Aug 8 00:48:03.458104986 (15h8m3.047449973s) [upload=15670]: Starting upload Aug 8 00:48:03.458166774 (15h8m3.047511761s) [upload=15670]: Done starting upload Aug 8 00:48:03.458226910 (15h8m3.047571887s) [upload=15670]: Uploading segments... Aug 8 00:48:03.458331268 (15h8m3.047676255s) [upload=15670]: Got next segment. Inline: true Aug 8 00:48:03.458336880 (15h8m3.047681857s) [upload=15670]: Waiting for error group managing segments... Aug 8 00:48:03.458383799 (15h8m3.047728786s) [upload=15670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:48:03.458388839 (15h8m3.047733827s) [upload=15670]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:48:03.458391154 (15h8m3.047736141s) [upload=15670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:48:03.756480234 (15h8m3.345825221s) [upload=15670]: Done uploading segments... Aug 8 00:48:03.756500813 (15h8m3.3458458s) [upload=15670]: Upload finished. err: <nil> Aug 8 00:49:01.779484346 (15h9m1.368829333s) [upload=15683]: Starting upload Aug 8 00:49:01.779541455 (15h9m1.368886442s) [upload=15683]: Done starting upload Aug 8 00:49:01.779591250 (15h9m1.368936238s) [upload=15683]: Uploading segments... Aug 8 00:49:01.779632399 (15h9m1.368977386s) [upload=15683]: Got next segment. Inline: true Aug 8 00:49:01.779637188 (15h9m1.368982185s) [upload=15683]: Waiting for error group managing segments... Aug 8 00:49:01.779673187 (15h9m1.369018174s) [upload=15683]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:49:01.779677375 (15h9m1.369022362s) [upload=15683]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:49:01.779680040 (15h9m1.369025027s) [upload=15683]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:49:02.060683371 (15h9m1.650028378s) [upload=15683]: Done uploading segments... Aug 8 00:49:02.060710513 (15h9m1.6500555s) [upload=15683]: Upload finished. err: <nil> Aug 8 00:50:06.726842474 (15h10m6.316187461s) [upload=15705]: Starting upload Aug 8 00:50:06.726885055 (15h10m6.316230032s) [upload=15705]: Done starting upload Aug 8 00:50:06.726932025 (15h10m6.316277012s) [upload=15705]: Uploading segments... Aug 8 00:50:06.726962704 (15h10m6.316307691s) [upload=15705]: Got next segment. Inline: true Aug 8 00:50:06.726966831 (15h10m6.316311808s) [upload=15705]: Waiting for error group managing segments... Aug 8 00:50:06.726999844 (15h10m6.316344831s) [upload=15705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:50:06.727003823 (15h10m6.31634881s) [upload=15705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:50:06.727006187 (15h10m6.316351164s) [upload=15705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:50:07.002622120 (15h10m6.591967127s) [upload=15705]: Done uploading segments... Aug 8 00:50:07.002639403 (15h10m6.59198438s) [upload=15705]: Upload finished. err: <nil> Aug 8 00:51:18.800365702 (15h11m18.389710689s) [upload=15723]: Starting upload Aug 8 00:51:18.800416448 (15h11m18.389761435s) [upload=15723]: Done starting upload Aug 8 00:51:18.800490300 (15h11m18.389835277s) [upload=15723]: Uploading segments... Aug 8 00:51:18.800624426 (15h11m18.389969413s) [upload=15723]: Got next segment. Inline: true Aug 8 00:51:18.800629807 (15h11m18.389974794s) [upload=15723]: Waiting for error group managing segments... Aug 8 00:51:18.800677268 (15h11m18.390022255s) [upload=15723]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:51:18.800682698 (15h11m18.390027685s) [upload=15723]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:51:18.800685353 (15h11m18.39003034s) [upload=15723]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:51:19.015137684 (15h11m18.604482671s) [upload=15723]: Done uploading segments... Aug 8 00:51:19.015155639 (15h11m18.604500616s) [upload=15723]: Upload finished. err: metaclient: Too Many Requests Duration: 2.285271879s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.......X...X............X....X........................] duration: 6.433646082s FAIL (2025-08-08 00:57:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837172821 (14h59m54.426517798s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837385949 (14h59m54.426730936s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837507802 (14h59m54.426852779s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837640145 (14h59m54.426985122s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837819459 (14h59m54.427164446s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837962502 (14h59m54.427307479s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838112790 (14h59m54.427457767s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838238691 (14h59m54.427583678s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838393447 (14h59m54.427738434s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838490402 (14h59m54.427835389s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838660288 (14h59m54.428005275s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838681548 (14h59m54.428026525s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838866312 (14h59m54.428211299s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838893975 (14h59m54.428238962s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.839056986 (14h59m54.428401974s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:55.935384249 (14h59m55.524729246s) [upload=15536]: Next returned nil segment Aug 8 00:39:55.935402815 (14h59m55.524747802s) [upload=15536]: Waiting for error group managing segments... Aug 8 00:39:56.226788010 (14h59m55.816132997s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.265710491 (14h59m55.855055478s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.326363980 (14h59m55.915708967s) [noise=true , address=64.49.13.169:4709 , piece_id=PIPH24KH4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773039652 (14h59m56.362384679s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773213735 (14h59m56.362558732s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.782851711 (14h59m56.372196698s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854376586 (14h59m56.443721573s) [noise=true , address=104.36.82.46:4484 , piece_id=BMHSO2DIXVRRHUZRHWUQ24TXMMTRMVX34SVDUUUMRMKPSUQG3APQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854703231 (14h59m56.444048218s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.914818370 (14h59m56.504163357s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.915843802 (14h59m56.505188849s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.942005621 (14h59m56.531350608s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.981103286 (14h59m56.570448273s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.095564053 (14h59m56.68490904s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216182613 (14h59m56.8055276s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216206869 (14h59m56.805551856s) [seg_pos={0 0}, upload=15536]: Segment reached optimal threshold of 14 pieces. Aug 8 00:39:57.216406632 (14h59m56.805751619s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bN25eEEjxCWKJSr8qj7moDJxArgfsjzPTdBcLdfvoeyxiqPpc): protocol: expected piece hash; context canceled; EOF Aug 8 00:39:57.216460465 (14h59m56.805805452s) [noise=true , address=104.36.82.5:4869 , piece_id=JBCDCG676BBV6U2GAVWDQIWYXKKQGW2DKK6MDY74LNPGP4K3CTQA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection Aug 8 00:39:57.216644467 (14h59m56.805989454s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD): protocol: read tcp 188.166.225.170:50728->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216665567 (14h59m56.806010554s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: read tcp 188.166.225.170:45052->104.36.82.61:4383: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216679534 (14h59m56.806024521s) [seg_pos={0 0}, upload=15536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:39:57.216739699 (14h59m56.806084686s) [upload=15536]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:39:57.216743286 (14h59m56.806088273s) [upload=15536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:39:57.516798834 (14h59m57.106143821s) [upload=15536]: Done uploading segments... Aug 8 00:39:57.516817129 (14h59m57.106162116s) [upload=15536]: Upload finished. err: <nil> Aug 8 00:40:09.579830255 (15h0m9.169175242s) [upload=15541]: Starting upload Aug 8 00:40:09.579877025 (15h0m9.169222012s) [upload=15541]: Done starting upload Aug 8 00:40:09.579941739 (15h0m9.169286726s) [upload=15541]: Uploading segments... Aug 8 00:40:09.580164405 (15h0m9.169509432s) [upload=15541]: Got next segment. Inline: false Aug 8 00:40:09.580219460 (15h0m9.169564477s) [upload=15541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:40:09.580251472 (15h0m9.169596489s) [upload=15541]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:40:09.797368422 (15h0m9.386713409s) [upload=15541]: Next segment err: metaclient: Too Many Requests Aug 8 00:40:09.797398149 (15h0m9.386743136s) [upload=15541]: Done uploading segments... Aug 8 00:40:09.797473614 (15h0m9.386818601s) [upload=15541]: Upload finished. err: metaclient: Too Many Requests Duration: 6.43367029s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........X.........................................X.......] duration: 1.838745151s FAIL (2025-08-08 00:56:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== stItem_ObjectCommit Aug 8 00:42:17.713078039 (15h2m17.302423046s) [upload=15574]: Done uploading segments... Aug 8 00:42:17.713100702 (15h2m17.30244574s) [upload=15574]: Upload finished. err: <nil> Aug 8 00:42:36.033866449 (15h2m35.623211436s) [upload=15582]: Starting upload Aug 8 00:42:36.033918057 (15h2m35.623263044s) [upload=15582]: Done starting upload Aug 8 00:42:36.033968163 (15h2m35.623313141s) [upload=15582]: Uploading segments... Aug 8 00:42:36.033998922 (15h2m35.623343909s) [upload=15582]: Got next segment. Inline: true Aug 8 00:42:36.034002399 (15h2m35.623347386s) [upload=15582]: Waiting for error group managing segments... Aug 8 00:42:36.034039881 (15h2m35.623384868s) [upload=15582]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:42:36.034043698 (15h2m35.623388685s) [upload=15582]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:42:36.034046063 (15h2m35.62339105s) [upload=15582]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:42:36.326408369 (15h2m35.915753356s) [upload=15582]: Done uploading segments... Aug 8 00:42:36.326427806 (15h2m35.915772823s) [upload=15582]: Upload finished. err: <nil> Aug 8 00:43:50.785976974 (15h3m50.375321961s) [upload=15600]: Starting upload Aug 8 00:43:50.786021608 (15h3m50.375366585s) [upload=15600]: Done starting upload Aug 8 00:43:50.786068328 (15h3m50.375413315s) [upload=15600]: Uploading segments... Aug 8 00:43:50.786096292 (15h3m50.375441269s) [upload=15600]: Got next segment. Inline: true Aug 8 00:43:50.786099928 (15h3m50.375444915s) [upload=15600]: Waiting for error group managing segments... Aug 8 00:43:50.786133022 (15h3m50.375478009s) [upload=15600]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:43:50.786136840 (15h3m50.375481817s) [upload=15600]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:43:50.786139374 (15h3m50.375484351s) [upload=15600]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:43:51.071276196 (15h3m50.660621183s) [upload=15600]: Done uploading segments... Aug 8 00:43:51.071294140 (15h3m50.660639127s) [upload=15600]: Upload finished. err: <nil> Aug 8 00:44:38.932060864 (15h4m38.521405851s) [upload=15614]: Starting upload Aug 8 00:44:38.932105570 (15h4m38.521450547s) [upload=15614]: Done starting upload Aug 8 00:44:38.932150475 (15h4m38.521495462s) [upload=15614]: Uploading segments... Aug 8 00:44:38.932180272 (15h4m38.521525249s) [upload=15614]: Got next segment. Inline: true Aug 8 00:44:38.932183829 (15h4m38.521528816s) [upload=15614]: Waiting for error group managing segments... Aug 8 00:44:38.932214318 (15h4m38.521559305s) [upload=15614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:44:38.932218205 (15h4m38.521563192s) [upload=15614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:44:38.932220419 (15h4m38.521565396s) [upload=15614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:44:39.257010779 (15h4m38.846355757s) [upload=15614]: Done uploading segments... Aug 8 00:44:39.257027903 (15h4m38.84637289s) [upload=15614]: Upload finished. err: <nil> Aug 8 00:44:55.671929119 (15h4m55.261274106s) [upload=15619]: Starting upload Aug 8 00:44:55.671974105 (15h4m55.261319092s) [upload=15619]: Done starting upload Aug 8 00:44:55.672033319 (15h4m55.261378296s) [upload=15619]: Uploading segments... Aug 8 00:44:55.672073296 (15h4m55.261418273s) [upload=15619]: Got next segment. Inline: true Aug 8 00:44:55.672077514 (15h4m55.261422501s) [upload=15619]: Waiting for error group managing segments... Aug 8 00:44:55.672111548 (15h4m55.261456535s) [upload=15619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:44:55.672115085 (15h4m55.261460072s) [upload=15619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:44:55.672117520 (15h4m55.261462507s) [upload=15619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:44:55.946154002 (15h4m55.535498989s) [upload=15619]: Done uploading segments... Aug 8 00:44:55.946180583 (15h4m55.53552557s) [upload=15619]: Upload finished. err: <nil> Aug 8 00:45:51.335687032 (15h5m50.925032019s) [upload=15634]: Starting upload Aug 8 00:45:51.335741947 (15h5m50.925086934s) [upload=15634]: Done starting upload Aug 8 00:45:51.335838182 (15h5m50.925183169s) [upload=15634]: Uploading segments... Aug 8 00:45:51.335870213 (15h5m50.9252152s) [upload=15634]: Got next segment. Inline: true Aug 8 00:45:51.335873930 (15h5m50.925218917s) [upload=15634]: Waiting for error group managing segments... Aug 8 00:45:51.335917033 (15h5m50.92526202s) [upload=15634]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:45:51.335921060 (15h5m50.925266047s) [upload=15634]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:45:51.335923345 (15h5m50.925268332s) [upload=15634]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:45:51.622624488 (15h5m51.211969475s) [upload=15634]: Done uploading segments... Aug 8 00:45:51.622650428 (15h5m51.211995415s) [upload=15634]: Upload finished. err: <nil> Aug 8 00:46:56.158030766 (15h6m55.747375753s) [upload=15655]: Starting upload Aug 8 00:46:56.158075261 (15h6m55.747420248s) [upload=15655]: Done starting upload Aug 8 00:46:56.158127792 (15h6m55.747472779s) [upload=15655]: Uploading segments... Aug 8 00:46:56.158159252 (15h6m55.747504229s) [upload=15655]: Got next segment. Inline: true Aug 8 00:46:56.158162568 (15h6m55.747507555s) [upload=15655]: Waiting for error group managing segments... Aug 8 00:46:56.158193087 (15h6m55.747538074s) [upload=15655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:46:56.158196653 (15h6m55.74754164s) [upload=15655]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:46:56.158199098 (15h6m55.747544075s) [upload=15655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:46:56.421913161 (15h6m56.011258168s) [upload=15655]: Done uploading segments... Aug 8 00:46:56.421936716 (15h6m56.011281703s) [upload=15655]: Upload finished. err: <nil> Aug 8 00:48:03.458104986 (15h8m3.047449973s) [upload=15670]: Starting upload Aug 8 00:48:03.458166774 (15h8m3.047511761s) [upload=15670]: Done starting upload Aug 8 00:48:03.458226910 (15h8m3.047571887s) [upload=15670]: Uploading segments... Aug 8 00:48:03.458331268 (15h8m3.047676255s) [upload=15670]: Got next segment. Inline: true Aug 8 00:48:03.458336880 (15h8m3.047681857s) [upload=15670]: Waiting for error group managing segments... Aug 8 00:48:03.458383799 (15h8m3.047728786s) [upload=15670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:48:03.458388839 (15h8m3.047733827s) [upload=15670]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:48:03.458391154 (15h8m3.047736141s) [upload=15670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:48:03.756480234 (15h8m3.345825221s) [upload=15670]: Done uploading segments... Aug 8 00:48:03.756500813 (15h8m3.3458458s) [upload=15670]: Upload finished. err: <nil> Aug 8 00:49:01.779484346 (15h9m1.368829333s) [upload=15683]: Starting upload Aug 8 00:49:01.779541455 (15h9m1.368886442s) [upload=15683]: Done starting upload Aug 8 00:49:01.779591250 (15h9m1.368936238s) [upload=15683]: Uploading segments... Aug 8 00:49:01.779632399 (15h9m1.368977386s) [upload=15683]: Got next segment. Inline: true Aug 8 00:49:01.779637188 (15h9m1.368982185s) [upload=15683]: Waiting for error group managing segments... Aug 8 00:49:01.779673187 (15h9m1.369018174s) [upload=15683]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:49:01.779677375 (15h9m1.369022362s) [upload=15683]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:49:01.779680040 (15h9m1.369025027s) [upload=15683]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:49:02.060683371 (15h9m1.650028378s) [upload=15683]: Done uploading segments... Aug 8 00:49:02.060710513 (15h9m1.6500555s) [upload=15683]: Upload finished. err: <nil> Aug 8 00:50:06.726842474 (15h10m6.316187461s) [upload=15705]: Starting upload Aug 8 00:50:06.726885055 (15h10m6.316230032s) [upload=15705]: Done starting upload Aug 8 00:50:06.726932025 (15h10m6.316277012s) [upload=15705]: Uploading segments... Aug 8 00:50:06.726962704 (15h10m6.316307691s) [upload=15705]: Got next segment. Inline: true Aug 8 00:50:06.726966831 (15h10m6.316311808s) [upload=15705]: Waiting for error group managing segments... Aug 8 00:50:06.726999844 (15h10m6.316344831s) [upload=15705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:50:06.727003823 (15h10m6.31634881s) [upload=15705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:50:06.727006187 (15h10m6.316351164s) [upload=15705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:50:07.002622120 (15h10m6.591967127s) [upload=15705]: Done uploading segments... Aug 8 00:50:07.002639403 (15h10m6.59198438s) [upload=15705]: Upload finished. err: <nil> Aug 8 00:51:18.800365702 (15h11m18.389710689s) [upload=15723]: Starting upload Aug 8 00:51:18.800416448 (15h11m18.389761435s) [upload=15723]: Done starting upload Aug 8 00:51:18.800490300 (15h11m18.389835277s) [upload=15723]: Uploading segments... Aug 8 00:51:18.800624426 (15h11m18.389969413s) [upload=15723]: Got next segment. Inline: true Aug 8 00:51:18.800629807 (15h11m18.389974794s) [upload=15723]: Waiting for error group managing segments... Aug 8 00:51:18.800677268 (15h11m18.390022255s) [upload=15723]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:51:18.800682698 (15h11m18.390027685s) [upload=15723]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:51:18.800685353 (15h11m18.39003034s) [upload=15723]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:51:19.015137684 (15h11m18.604482671s) [upload=15723]: Done uploading segments... Aug 8 00:51:19.015155639 (15h11m18.604500616s) [upload=15723]: Upload finished. err: metaclient: Too Many Requests Duration: 1.838773968s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.......X...X............X....X.......................] duration: 6.123959949s FAIL (2025-08-08 00:56:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837172821 (14h59m54.426517798s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837385949 (14h59m54.426730936s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837507802 (14h59m54.426852779s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837640145 (14h59m54.426985122s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837819459 (14h59m54.427164446s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837962502 (14h59m54.427307479s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838112790 (14h59m54.427457767s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838238691 (14h59m54.427583678s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838393447 (14h59m54.427738434s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838490402 (14h59m54.427835389s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838660288 (14h59m54.428005275s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838681548 (14h59m54.428026525s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838866312 (14h59m54.428211299s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838893975 (14h59m54.428238962s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.839056986 (14h59m54.428401974s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:55.935384249 (14h59m55.524729246s) [upload=15536]: Next returned nil segment Aug 8 00:39:55.935402815 (14h59m55.524747802s) [upload=15536]: Waiting for error group managing segments... Aug 8 00:39:56.226788010 (14h59m55.816132997s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.265710491 (14h59m55.855055478s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.326363980 (14h59m55.915708967s) [noise=true , address=64.49.13.169:4709 , piece_id=PIPH24KH4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773039652 (14h59m56.362384679s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773213735 (14h59m56.362558732s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.782851711 (14h59m56.372196698s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854376586 (14h59m56.443721573s) [noise=true , address=104.36.82.46:4484 , piece_id=BMHSO2DIXVRRHUZRHWUQ24TXMMTRMVX34SVDUUUMRMKPSUQG3APQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854703231 (14h59m56.444048218s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.914818370 (14h59m56.504163357s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.915843802 (14h59m56.505188849s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.942005621 (14h59m56.531350608s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.981103286 (14h59m56.570448273s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.095564053 (14h59m56.68490904s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216182613 (14h59m56.8055276s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216206869 (14h59m56.805551856s) [seg_pos={0 0}, upload=15536]: Segment reached optimal threshold of 14 pieces. Aug 8 00:39:57.216406632 (14h59m56.805751619s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bN25eEEjxCWKJSr8qj7moDJxArgfsjzPTdBcLdfvoeyxiqPpc): protocol: expected piece hash; context canceled; EOF Aug 8 00:39:57.216460465 (14h59m56.805805452s) [noise=true , address=104.36.82.5:4869 , piece_id=JBCDCG676BBV6U2GAVWDQIWYXKKQGW2DKK6MDY74LNPGP4K3CTQA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection Aug 8 00:39:57.216644467 (14h59m56.805989454s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD): protocol: read tcp 188.166.225.170:50728->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216665567 (14h59m56.806010554s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: read tcp 188.166.225.170:45052->104.36.82.61:4383: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216679534 (14h59m56.806024521s) [seg_pos={0 0}, upload=15536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:39:57.216739699 (14h59m56.806084686s) [upload=15536]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:39:57.216743286 (14h59m56.806088273s) [upload=15536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:39:57.516798834 (14h59m57.106143821s) [upload=15536]: Done uploading segments... Aug 8 00:39:57.516817129 (14h59m57.106162116s) [upload=15536]: Upload finished. err: <nil> Aug 8 00:40:09.579830255 (15h0m9.169175242s) [upload=15541]: Starting upload Aug 8 00:40:09.579877025 (15h0m9.169222012s) [upload=15541]: Done starting upload Aug 8 00:40:09.579941739 (15h0m9.169286726s) [upload=15541]: Uploading segments... Aug 8 00:40:09.580164405 (15h0m9.169509432s) [upload=15541]: Got next segment. Inline: false Aug 8 00:40:09.580219460 (15h0m9.169564477s) [upload=15541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:40:09.580251472 (15h0m9.169596489s) [upload=15541]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:40:09.797368422 (15h0m9.386713409s) [upload=15541]: Next segment err: metaclient: Too Many Requests Aug 8 00:40:09.797398149 (15h0m9.386743136s) [upload=15541]: Done uploading segments... Aug 8 00:40:09.797473614 (15h0m9.386818601s) [upload=15541]: Upload finished. err: metaclient: Too Many Requests Duration: 6.123984537s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........X.........................................X......] duration: 2.413450705s FAIL (2025-08-08 00:55:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== stItem_ObjectCommit Aug 8 00:42:17.713078039 (15h2m17.302423046s) [upload=15574]: Done uploading segments... Aug 8 00:42:17.713100702 (15h2m17.30244574s) [upload=15574]: Upload finished. err: <nil> Aug 8 00:42:36.033866449 (15h2m35.623211436s) [upload=15582]: Starting upload Aug 8 00:42:36.033918057 (15h2m35.623263044s) [upload=15582]: Done starting upload Aug 8 00:42:36.033968163 (15h2m35.623313141s) [upload=15582]: Uploading segments... Aug 8 00:42:36.033998922 (15h2m35.623343909s) [upload=15582]: Got next segment. Inline: true Aug 8 00:42:36.034002399 (15h2m35.623347386s) [upload=15582]: Waiting for error group managing segments... Aug 8 00:42:36.034039881 (15h2m35.623384868s) [upload=15582]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:42:36.034043698 (15h2m35.623388685s) [upload=15582]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:42:36.034046063 (15h2m35.62339105s) [upload=15582]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:42:36.326408369 (15h2m35.915753356s) [upload=15582]: Done uploading segments... Aug 8 00:42:36.326427806 (15h2m35.915772823s) [upload=15582]: Upload finished. err: <nil> Aug 8 00:43:50.785976974 (15h3m50.375321961s) [upload=15600]: Starting upload Aug 8 00:43:50.786021608 (15h3m50.375366585s) [upload=15600]: Done starting upload Aug 8 00:43:50.786068328 (15h3m50.375413315s) [upload=15600]: Uploading segments... Aug 8 00:43:50.786096292 (15h3m50.375441269s) [upload=15600]: Got next segment. Inline: true Aug 8 00:43:50.786099928 (15h3m50.375444915s) [upload=15600]: Waiting for error group managing segments... Aug 8 00:43:50.786133022 (15h3m50.375478009s) [upload=15600]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:43:50.786136840 (15h3m50.375481817s) [upload=15600]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:43:50.786139374 (15h3m50.375484351s) [upload=15600]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:43:51.071276196 (15h3m50.660621183s) [upload=15600]: Done uploading segments... Aug 8 00:43:51.071294140 (15h3m50.660639127s) [upload=15600]: Upload finished. err: <nil> Aug 8 00:44:38.932060864 (15h4m38.521405851s) [upload=15614]: Starting upload Aug 8 00:44:38.932105570 (15h4m38.521450547s) [upload=15614]: Done starting upload Aug 8 00:44:38.932150475 (15h4m38.521495462s) [upload=15614]: Uploading segments... Aug 8 00:44:38.932180272 (15h4m38.521525249s) [upload=15614]: Got next segment. Inline: true Aug 8 00:44:38.932183829 (15h4m38.521528816s) [upload=15614]: Waiting for error group managing segments... Aug 8 00:44:38.932214318 (15h4m38.521559305s) [upload=15614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:44:38.932218205 (15h4m38.521563192s) [upload=15614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:44:38.932220419 (15h4m38.521565396s) [upload=15614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:44:39.257010779 (15h4m38.846355757s) [upload=15614]: Done uploading segments... Aug 8 00:44:39.257027903 (15h4m38.84637289s) [upload=15614]: Upload finished. err: <nil> Aug 8 00:44:55.671929119 (15h4m55.261274106s) [upload=15619]: Starting upload Aug 8 00:44:55.671974105 (15h4m55.261319092s) [upload=15619]: Done starting upload Aug 8 00:44:55.672033319 (15h4m55.261378296s) [upload=15619]: Uploading segments... Aug 8 00:44:55.672073296 (15h4m55.261418273s) [upload=15619]: Got next segment. Inline: true Aug 8 00:44:55.672077514 (15h4m55.261422501s) [upload=15619]: Waiting for error group managing segments... Aug 8 00:44:55.672111548 (15h4m55.261456535s) [upload=15619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:44:55.672115085 (15h4m55.261460072s) [upload=15619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:44:55.672117520 (15h4m55.261462507s) [upload=15619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:44:55.946154002 (15h4m55.535498989s) [upload=15619]: Done uploading segments... Aug 8 00:44:55.946180583 (15h4m55.53552557s) [upload=15619]: Upload finished. err: <nil> Aug 8 00:45:51.335687032 (15h5m50.925032019s) [upload=15634]: Starting upload Aug 8 00:45:51.335741947 (15h5m50.925086934s) [upload=15634]: Done starting upload Aug 8 00:45:51.335838182 (15h5m50.925183169s) [upload=15634]: Uploading segments... Aug 8 00:45:51.335870213 (15h5m50.9252152s) [upload=15634]: Got next segment. Inline: true Aug 8 00:45:51.335873930 (15h5m50.925218917s) [upload=15634]: Waiting for error group managing segments... Aug 8 00:45:51.335917033 (15h5m50.92526202s) [upload=15634]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:45:51.335921060 (15h5m50.925266047s) [upload=15634]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:45:51.335923345 (15h5m50.925268332s) [upload=15634]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:45:51.622624488 (15h5m51.211969475s) [upload=15634]: Done uploading segments... Aug 8 00:45:51.622650428 (15h5m51.211995415s) [upload=15634]: Upload finished. err: <nil> Aug 8 00:46:56.158030766 (15h6m55.747375753s) [upload=15655]: Starting upload Aug 8 00:46:56.158075261 (15h6m55.747420248s) [upload=15655]: Done starting upload Aug 8 00:46:56.158127792 (15h6m55.747472779s) [upload=15655]: Uploading segments... Aug 8 00:46:56.158159252 (15h6m55.747504229s) [upload=15655]: Got next segment. Inline: true Aug 8 00:46:56.158162568 (15h6m55.747507555s) [upload=15655]: Waiting for error group managing segments... Aug 8 00:46:56.158193087 (15h6m55.747538074s) [upload=15655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:46:56.158196653 (15h6m55.74754164s) [upload=15655]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:46:56.158199098 (15h6m55.747544075s) [upload=15655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:46:56.421913161 (15h6m56.011258168s) [upload=15655]: Done uploading segments... Aug 8 00:46:56.421936716 (15h6m56.011281703s) [upload=15655]: Upload finished. err: <nil> Aug 8 00:48:03.458104986 (15h8m3.047449973s) [upload=15670]: Starting upload Aug 8 00:48:03.458166774 (15h8m3.047511761s) [upload=15670]: Done starting upload Aug 8 00:48:03.458226910 (15h8m3.047571887s) [upload=15670]: Uploading segments... Aug 8 00:48:03.458331268 (15h8m3.047676255s) [upload=15670]: Got next segment. Inline: true Aug 8 00:48:03.458336880 (15h8m3.047681857s) [upload=15670]: Waiting for error group managing segments... Aug 8 00:48:03.458383799 (15h8m3.047728786s) [upload=15670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:48:03.458388839 (15h8m3.047733827s) [upload=15670]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:48:03.458391154 (15h8m3.047736141s) [upload=15670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:48:03.756480234 (15h8m3.345825221s) [upload=15670]: Done uploading segments... Aug 8 00:48:03.756500813 (15h8m3.3458458s) [upload=15670]: Upload finished. err: <nil> Aug 8 00:49:01.779484346 (15h9m1.368829333s) [upload=15683]: Starting upload Aug 8 00:49:01.779541455 (15h9m1.368886442s) [upload=15683]: Done starting upload Aug 8 00:49:01.779591250 (15h9m1.368936238s) [upload=15683]: Uploading segments... Aug 8 00:49:01.779632399 (15h9m1.368977386s) [upload=15683]: Got next segment. Inline: true Aug 8 00:49:01.779637188 (15h9m1.368982185s) [upload=15683]: Waiting for error group managing segments... Aug 8 00:49:01.779673187 (15h9m1.369018174s) [upload=15683]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:49:01.779677375 (15h9m1.369022362s) [upload=15683]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:49:01.779680040 (15h9m1.369025027s) [upload=15683]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:49:02.060683371 (15h9m1.650028378s) [upload=15683]: Done uploading segments... Aug 8 00:49:02.060710513 (15h9m1.6500555s) [upload=15683]: Upload finished. err: <nil> Aug 8 00:50:06.726842474 (15h10m6.316187461s) [upload=15705]: Starting upload Aug 8 00:50:06.726885055 (15h10m6.316230032s) [upload=15705]: Done starting upload Aug 8 00:50:06.726932025 (15h10m6.316277012s) [upload=15705]: Uploading segments... Aug 8 00:50:06.726962704 (15h10m6.316307691s) [upload=15705]: Got next segment. Inline: true Aug 8 00:50:06.726966831 (15h10m6.316311808s) [upload=15705]: Waiting for error group managing segments... Aug 8 00:50:06.726999844 (15h10m6.316344831s) [upload=15705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:50:06.727003823 (15h10m6.31634881s) [upload=15705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:50:06.727006187 (15h10m6.316351164s) [upload=15705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:50:07.002622120 (15h10m6.591967127s) [upload=15705]: Done uploading segments... Aug 8 00:50:07.002639403 (15h10m6.59198438s) [upload=15705]: Upload finished. err: <nil> Aug 8 00:51:18.800365702 (15h11m18.389710689s) [upload=15723]: Starting upload Aug 8 00:51:18.800416448 (15h11m18.389761435s) [upload=15723]: Done starting upload Aug 8 00:51:18.800490300 (15h11m18.389835277s) [upload=15723]: Uploading segments... Aug 8 00:51:18.800624426 (15h11m18.389969413s) [upload=15723]: Got next segment. Inline: true Aug 8 00:51:18.800629807 (15h11m18.389974794s) [upload=15723]: Waiting for error group managing segments... Aug 8 00:51:18.800677268 (15h11m18.390022255s) [upload=15723]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:51:18.800682698 (15h11m18.390027685s) [upload=15723]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:51:18.800685353 (15h11m18.39003034s) [upload=15723]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:51:19.015137684 (15h11m18.604482671s) [upload=15723]: Done uploading segments... Aug 8 00:51:19.015155639 (15h11m18.604500616s) [upload=15723]: Upload finished. err: metaclient: Too Many Requests Duration: 2.413483016s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X.......X...X............X....X......................] duration: 6.201210323s FAIL (2025-08-08 00:55:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837172821 (14h59m54.426517798s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837385949 (14h59m54.426730936s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837507802 (14h59m54.426852779s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837640145 (14h59m54.426985122s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837819459 (14h59m54.427164446s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837962502 (14h59m54.427307479s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838112790 (14h59m54.427457767s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838238691 (14h59m54.427583678s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838393447 (14h59m54.427738434s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838490402 (14h59m54.427835389s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838660288 (14h59m54.428005275s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838681548 (14h59m54.428026525s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838866312 (14h59m54.428211299s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838893975 (14h59m54.428238962s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.839056986 (14h59m54.428401974s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:55.935384249 (14h59m55.524729246s) [upload=15536]: Next returned nil segment Aug 8 00:39:55.935402815 (14h59m55.524747802s) [upload=15536]: Waiting for error group managing segments... Aug 8 00:39:56.226788010 (14h59m55.816132997s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.265710491 (14h59m55.855055478s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.326363980 (14h59m55.915708967s) [noise=true , address=64.49.13.169:4709 , piece_id=PIPH24KH4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773039652 (14h59m56.362384679s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773213735 (14h59m56.362558732s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.782851711 (14h59m56.372196698s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854376586 (14h59m56.443721573s) [noise=true , address=104.36.82.46:4484 , piece_id=BMHSO2DIXVRRHUZRHWUQ24TXMMTRMVX34SVDUUUMRMKPSUQG3APQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854703231 (14h59m56.444048218s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.914818370 (14h59m56.504163357s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.915843802 (14h59m56.505188849s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.942005621 (14h59m56.531350608s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.981103286 (14h59m56.570448273s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.095564053 (14h59m56.68490904s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216182613 (14h59m56.8055276s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216206869 (14h59m56.805551856s) [seg_pos={0 0}, upload=15536]: Segment reached optimal threshold of 14 pieces. Aug 8 00:39:57.216406632 (14h59m56.805751619s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bN25eEEjxCWKJSr8qj7moDJxArgfsjzPTdBcLdfvoeyxiqPpc): protocol: expected piece hash; context canceled; EOF Aug 8 00:39:57.216460465 (14h59m56.805805452s) [noise=true , address=104.36.82.5:4869 , piece_id=JBCDCG676BBV6U2GAVWDQIWYXKKQGW2DKK6MDY74LNPGP4K3CTQA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection Aug 8 00:39:57.216644467 (14h59m56.805989454s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD): protocol: read tcp 188.166.225.170:50728->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216665567 (14h59m56.806010554s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: read tcp 188.166.225.170:45052->104.36.82.61:4383: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216679534 (14h59m56.806024521s) [seg_pos={0 0}, upload=15536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:39:57.216739699 (14h59m56.806084686s) [upload=15536]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:39:57.216743286 (14h59m56.806088273s) [upload=15536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:39:57.516798834 (14h59m57.106143821s) [upload=15536]: Done uploading segments... Aug 8 00:39:57.516817129 (14h59m57.106162116s) [upload=15536]: Upload finished. err: <nil> Aug 8 00:40:09.579830255 (15h0m9.169175242s) [upload=15541]: Starting upload Aug 8 00:40:09.579877025 (15h0m9.169222012s) [upload=15541]: Done starting upload Aug 8 00:40:09.579941739 (15h0m9.169286726s) [upload=15541]: Uploading segments... Aug 8 00:40:09.580164405 (15h0m9.169509432s) [upload=15541]: Got next segment. Inline: false Aug 8 00:40:09.580219460 (15h0m9.169564477s) [upload=15541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:40:09.580251472 (15h0m9.169596489s) [upload=15541]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:40:09.797368422 (15h0m9.386713409s) [upload=15541]: Next segment err: metaclient: Too Many Requests Aug 8 00:40:09.797398149 (15h0m9.386743136s) [upload=15541]: Done uploading segments... Aug 8 00:40:09.797473614 (15h0m9.386818601s) [upload=15541]: Upload finished. err: metaclient: Too Many Requests Duration: 6.201237827s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............X.........................................X.....] duration: 1.833409224s FAIL (2025-08-08 00:54:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== stItem_ObjectCommit Aug 8 00:42:17.713078039 (15h2m17.302423046s) [upload=15574]: Done uploading segments... Aug 8 00:42:17.713100702 (15h2m17.30244574s) [upload=15574]: Upload finished. err: <nil> Aug 8 00:42:36.033866449 (15h2m35.623211436s) [upload=15582]: Starting upload Aug 8 00:42:36.033918057 (15h2m35.623263044s) [upload=15582]: Done starting upload Aug 8 00:42:36.033968163 (15h2m35.623313141s) [upload=15582]: Uploading segments... Aug 8 00:42:36.033998922 (15h2m35.623343909s) [upload=15582]: Got next segment. Inline: true Aug 8 00:42:36.034002399 (15h2m35.623347386s) [upload=15582]: Waiting for error group managing segments... Aug 8 00:42:36.034039881 (15h2m35.623384868s) [upload=15582]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:42:36.034043698 (15h2m35.623388685s) [upload=15582]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:42:36.034046063 (15h2m35.62339105s) [upload=15582]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:42:36.326408369 (15h2m35.915753356s) [upload=15582]: Done uploading segments... Aug 8 00:42:36.326427806 (15h2m35.915772823s) [upload=15582]: Upload finished. err: <nil> Aug 8 00:43:50.785976974 (15h3m50.375321961s) [upload=15600]: Starting upload Aug 8 00:43:50.786021608 (15h3m50.375366585s) [upload=15600]: Done starting upload Aug 8 00:43:50.786068328 (15h3m50.375413315s) [upload=15600]: Uploading segments... Aug 8 00:43:50.786096292 (15h3m50.375441269s) [upload=15600]: Got next segment. Inline: true Aug 8 00:43:50.786099928 (15h3m50.375444915s) [upload=15600]: Waiting for error group managing segments... Aug 8 00:43:50.786133022 (15h3m50.375478009s) [upload=15600]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:43:50.786136840 (15h3m50.375481817s) [upload=15600]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:43:50.786139374 (15h3m50.375484351s) [upload=15600]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:43:51.071276196 (15h3m50.660621183s) [upload=15600]: Done uploading segments... Aug 8 00:43:51.071294140 (15h3m50.660639127s) [upload=15600]: Upload finished. err: <nil> Aug 8 00:44:38.932060864 (15h4m38.521405851s) [upload=15614]: Starting upload Aug 8 00:44:38.932105570 (15h4m38.521450547s) [upload=15614]: Done starting upload Aug 8 00:44:38.932150475 (15h4m38.521495462s) [upload=15614]: Uploading segments... Aug 8 00:44:38.932180272 (15h4m38.521525249s) [upload=15614]: Got next segment. Inline: true Aug 8 00:44:38.932183829 (15h4m38.521528816s) [upload=15614]: Waiting for error group managing segments... Aug 8 00:44:38.932214318 (15h4m38.521559305s) [upload=15614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:44:38.932218205 (15h4m38.521563192s) [upload=15614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:44:38.932220419 (15h4m38.521565396s) [upload=15614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:44:39.257010779 (15h4m38.846355757s) [upload=15614]: Done uploading segments... Aug 8 00:44:39.257027903 (15h4m38.84637289s) [upload=15614]: Upload finished. err: <nil> Aug 8 00:44:55.671929119 (15h4m55.261274106s) [upload=15619]: Starting upload Aug 8 00:44:55.671974105 (15h4m55.261319092s) [upload=15619]: Done starting upload Aug 8 00:44:55.672033319 (15h4m55.261378296s) [upload=15619]: Uploading segments... Aug 8 00:44:55.672073296 (15h4m55.261418273s) [upload=15619]: Got next segment. Inline: true Aug 8 00:44:55.672077514 (15h4m55.261422501s) [upload=15619]: Waiting for error group managing segments... Aug 8 00:44:55.672111548 (15h4m55.261456535s) [upload=15619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:44:55.672115085 (15h4m55.261460072s) [upload=15619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:44:55.672117520 (15h4m55.261462507s) [upload=15619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:44:55.946154002 (15h4m55.535498989s) [upload=15619]: Done uploading segments... Aug 8 00:44:55.946180583 (15h4m55.53552557s) [upload=15619]: Upload finished. err: <nil> Aug 8 00:45:51.335687032 (15h5m50.925032019s) [upload=15634]: Starting upload Aug 8 00:45:51.335741947 (15h5m50.925086934s) [upload=15634]: Done starting upload Aug 8 00:45:51.335838182 (15h5m50.925183169s) [upload=15634]: Uploading segments... Aug 8 00:45:51.335870213 (15h5m50.9252152s) [upload=15634]: Got next segment. Inline: true Aug 8 00:45:51.335873930 (15h5m50.925218917s) [upload=15634]: Waiting for error group managing segments... Aug 8 00:45:51.335917033 (15h5m50.92526202s) [upload=15634]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:45:51.335921060 (15h5m50.925266047s) [upload=15634]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:45:51.335923345 (15h5m50.925268332s) [upload=15634]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:45:51.622624488 (15h5m51.211969475s) [upload=15634]: Done uploading segments... Aug 8 00:45:51.622650428 (15h5m51.211995415s) [upload=15634]: Upload finished. err: <nil> Aug 8 00:46:56.158030766 (15h6m55.747375753s) [upload=15655]: Starting upload Aug 8 00:46:56.158075261 (15h6m55.747420248s) [upload=15655]: Done starting upload Aug 8 00:46:56.158127792 (15h6m55.747472779s) [upload=15655]: Uploading segments... Aug 8 00:46:56.158159252 (15h6m55.747504229s) [upload=15655]: Got next segment. Inline: true Aug 8 00:46:56.158162568 (15h6m55.747507555s) [upload=15655]: Waiting for error group managing segments... Aug 8 00:46:56.158193087 (15h6m55.747538074s) [upload=15655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:46:56.158196653 (15h6m55.74754164s) [upload=15655]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:46:56.158199098 (15h6m55.747544075s) [upload=15655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:46:56.421913161 (15h6m56.011258168s) [upload=15655]: Done uploading segments... Aug 8 00:46:56.421936716 (15h6m56.011281703s) [upload=15655]: Upload finished. err: <nil> Aug 8 00:48:03.458104986 (15h8m3.047449973s) [upload=15670]: Starting upload Aug 8 00:48:03.458166774 (15h8m3.047511761s) [upload=15670]: Done starting upload Aug 8 00:48:03.458226910 (15h8m3.047571887s) [upload=15670]: Uploading segments... Aug 8 00:48:03.458331268 (15h8m3.047676255s) [upload=15670]: Got next segment. Inline: true Aug 8 00:48:03.458336880 (15h8m3.047681857s) [upload=15670]: Waiting for error group managing segments... Aug 8 00:48:03.458383799 (15h8m3.047728786s) [upload=15670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:48:03.458388839 (15h8m3.047733827s) [upload=15670]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:48:03.458391154 (15h8m3.047736141s) [upload=15670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:48:03.756480234 (15h8m3.345825221s) [upload=15670]: Done uploading segments... Aug 8 00:48:03.756500813 (15h8m3.3458458s) [upload=15670]: Upload finished. err: <nil> Aug 8 00:49:01.779484346 (15h9m1.368829333s) [upload=15683]: Starting upload Aug 8 00:49:01.779541455 (15h9m1.368886442s) [upload=15683]: Done starting upload Aug 8 00:49:01.779591250 (15h9m1.368936238s) [upload=15683]: Uploading segments... Aug 8 00:49:01.779632399 (15h9m1.368977386s) [upload=15683]: Got next segment. Inline: true Aug 8 00:49:01.779637188 (15h9m1.368982185s) [upload=15683]: Waiting for error group managing segments... Aug 8 00:49:01.779673187 (15h9m1.369018174s) [upload=15683]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:49:01.779677375 (15h9m1.369022362s) [upload=15683]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:49:01.779680040 (15h9m1.369025027s) [upload=15683]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:49:02.060683371 (15h9m1.650028378s) [upload=15683]: Done uploading segments... Aug 8 00:49:02.060710513 (15h9m1.6500555s) [upload=15683]: Upload finished. err: <nil> Aug 8 00:50:06.726842474 (15h10m6.316187461s) [upload=15705]: Starting upload Aug 8 00:50:06.726885055 (15h10m6.316230032s) [upload=15705]: Done starting upload Aug 8 00:50:06.726932025 (15h10m6.316277012s) [upload=15705]: Uploading segments... Aug 8 00:50:06.726962704 (15h10m6.316307691s) [upload=15705]: Got next segment. Inline: true Aug 8 00:50:06.726966831 (15h10m6.316311808s) [upload=15705]: Waiting for error group managing segments... Aug 8 00:50:06.726999844 (15h10m6.316344831s) [upload=15705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:50:06.727003823 (15h10m6.31634881s) [upload=15705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:50:06.727006187 (15h10m6.316351164s) [upload=15705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:50:07.002622120 (15h10m6.591967127s) [upload=15705]: Done uploading segments... Aug 8 00:50:07.002639403 (15h10m6.59198438s) [upload=15705]: Upload finished. err: <nil> Aug 8 00:51:18.800365702 (15h11m18.389710689s) [upload=15723]: Starting upload Aug 8 00:51:18.800416448 (15h11m18.389761435s) [upload=15723]: Done starting upload Aug 8 00:51:18.800490300 (15h11m18.389835277s) [upload=15723]: Uploading segments... Aug 8 00:51:18.800624426 (15h11m18.389969413s) [upload=15723]: Got next segment. Inline: true Aug 8 00:51:18.800629807 (15h11m18.389974794s) [upload=15723]: Waiting for error group managing segments... Aug 8 00:51:18.800677268 (15h11m18.390022255s) [upload=15723]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:51:18.800682698 (15h11m18.390027685s) [upload=15723]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:51:18.800685353 (15h11m18.39003034s) [upload=15723]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:51:19.015137684 (15h11m18.604482671s) [upload=15723]: Done uploading segments... Aug 8 00:51:19.015155639 (15h11m18.604500616s) [upload=15723]: Upload finished. err: metaclient: Too Many Requests Duration: 1.83343894s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............X.........................................X....] duration: 1.88192584s FAIL (2025-08-08 00:54:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== stItem_ObjectCommit Aug 8 00:42:17.713078039 (15h2m17.302423046s) [upload=15574]: Done uploading segments... Aug 8 00:42:17.713100702 (15h2m17.30244574s) [upload=15574]: Upload finished. err: <nil> Aug 8 00:42:36.033866449 (15h2m35.623211436s) [upload=15582]: Starting upload Aug 8 00:42:36.033918057 (15h2m35.623263044s) [upload=15582]: Done starting upload Aug 8 00:42:36.033968163 (15h2m35.623313141s) [upload=15582]: Uploading segments... Aug 8 00:42:36.033998922 (15h2m35.623343909s) [upload=15582]: Got next segment. Inline: true Aug 8 00:42:36.034002399 (15h2m35.623347386s) [upload=15582]: Waiting for error group managing segments... Aug 8 00:42:36.034039881 (15h2m35.623384868s) [upload=15582]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:42:36.034043698 (15h2m35.623388685s) [upload=15582]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:42:36.034046063 (15h2m35.62339105s) [upload=15582]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:42:36.326408369 (15h2m35.915753356s) [upload=15582]: Done uploading segments... Aug 8 00:42:36.326427806 (15h2m35.915772823s) [upload=15582]: Upload finished. err: <nil> Aug 8 00:43:50.785976974 (15h3m50.375321961s) [upload=15600]: Starting upload Aug 8 00:43:50.786021608 (15h3m50.375366585s) [upload=15600]: Done starting upload Aug 8 00:43:50.786068328 (15h3m50.375413315s) [upload=15600]: Uploading segments... Aug 8 00:43:50.786096292 (15h3m50.375441269s) [upload=15600]: Got next segment. Inline: true Aug 8 00:43:50.786099928 (15h3m50.375444915s) [upload=15600]: Waiting for error group managing segments... Aug 8 00:43:50.786133022 (15h3m50.375478009s) [upload=15600]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:43:50.786136840 (15h3m50.375481817s) [upload=15600]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:43:50.786139374 (15h3m50.375484351s) [upload=15600]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:43:51.071276196 (15h3m50.660621183s) [upload=15600]: Done uploading segments... Aug 8 00:43:51.071294140 (15h3m50.660639127s) [upload=15600]: Upload finished. err: <nil> Aug 8 00:44:38.932060864 (15h4m38.521405851s) [upload=15614]: Starting upload Aug 8 00:44:38.932105570 (15h4m38.521450547s) [upload=15614]: Done starting upload Aug 8 00:44:38.932150475 (15h4m38.521495462s) [upload=15614]: Uploading segments... Aug 8 00:44:38.932180272 (15h4m38.521525249s) [upload=15614]: Got next segment. Inline: true Aug 8 00:44:38.932183829 (15h4m38.521528816s) [upload=15614]: Waiting for error group managing segments... Aug 8 00:44:38.932214318 (15h4m38.521559305s) [upload=15614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:44:38.932218205 (15h4m38.521563192s) [upload=15614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:44:38.932220419 (15h4m38.521565396s) [upload=15614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:44:39.257010779 (15h4m38.846355757s) [upload=15614]: Done uploading segments... Aug 8 00:44:39.257027903 (15h4m38.84637289s) [upload=15614]: Upload finished. err: <nil> Aug 8 00:44:55.671929119 (15h4m55.261274106s) [upload=15619]: Starting upload Aug 8 00:44:55.671974105 (15h4m55.261319092s) [upload=15619]: Done starting upload Aug 8 00:44:55.672033319 (15h4m55.261378296s) [upload=15619]: Uploading segments... Aug 8 00:44:55.672073296 (15h4m55.261418273s) [upload=15619]: Got next segment. Inline: true Aug 8 00:44:55.672077514 (15h4m55.261422501s) [upload=15619]: Waiting for error group managing segments... Aug 8 00:44:55.672111548 (15h4m55.261456535s) [upload=15619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:44:55.672115085 (15h4m55.261460072s) [upload=15619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:44:55.672117520 (15h4m55.261462507s) [upload=15619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:44:55.946154002 (15h4m55.535498989s) [upload=15619]: Done uploading segments... Aug 8 00:44:55.946180583 (15h4m55.53552557s) [upload=15619]: Upload finished. err: <nil> Aug 8 00:45:51.335687032 (15h5m50.925032019s) [upload=15634]: Starting upload Aug 8 00:45:51.335741947 (15h5m50.925086934s) [upload=15634]: Done starting upload Aug 8 00:45:51.335838182 (15h5m50.925183169s) [upload=15634]: Uploading segments... Aug 8 00:45:51.335870213 (15h5m50.9252152s) [upload=15634]: Got next segment. Inline: true Aug 8 00:45:51.335873930 (15h5m50.925218917s) [upload=15634]: Waiting for error group managing segments... Aug 8 00:45:51.335917033 (15h5m50.92526202s) [upload=15634]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:45:51.335921060 (15h5m50.925266047s) [upload=15634]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:45:51.335923345 (15h5m50.925268332s) [upload=15634]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:45:51.622624488 (15h5m51.211969475s) [upload=15634]: Done uploading segments... Aug 8 00:45:51.622650428 (15h5m51.211995415s) [upload=15634]: Upload finished. err: <nil> Aug 8 00:46:56.158030766 (15h6m55.747375753s) [upload=15655]: Starting upload Aug 8 00:46:56.158075261 (15h6m55.747420248s) [upload=15655]: Done starting upload Aug 8 00:46:56.158127792 (15h6m55.747472779s) [upload=15655]: Uploading segments... Aug 8 00:46:56.158159252 (15h6m55.747504229s) [upload=15655]: Got next segment. Inline: true Aug 8 00:46:56.158162568 (15h6m55.747507555s) [upload=15655]: Waiting for error group managing segments... Aug 8 00:46:56.158193087 (15h6m55.747538074s) [upload=15655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:46:56.158196653 (15h6m55.74754164s) [upload=15655]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:46:56.158199098 (15h6m55.747544075s) [upload=15655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:46:56.421913161 (15h6m56.011258168s) [upload=15655]: Done uploading segments... Aug 8 00:46:56.421936716 (15h6m56.011281703s) [upload=15655]: Upload finished. err: <nil> Aug 8 00:48:03.458104986 (15h8m3.047449973s) [upload=15670]: Starting upload Aug 8 00:48:03.458166774 (15h8m3.047511761s) [upload=15670]: Done starting upload Aug 8 00:48:03.458226910 (15h8m3.047571887s) [upload=15670]: Uploading segments... Aug 8 00:48:03.458331268 (15h8m3.047676255s) [upload=15670]: Got next segment. Inline: true Aug 8 00:48:03.458336880 (15h8m3.047681857s) [upload=15670]: Waiting for error group managing segments... Aug 8 00:48:03.458383799 (15h8m3.047728786s) [upload=15670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:48:03.458388839 (15h8m3.047733827s) [upload=15670]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:48:03.458391154 (15h8m3.047736141s) [upload=15670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:48:03.756480234 (15h8m3.345825221s) [upload=15670]: Done uploading segments... Aug 8 00:48:03.756500813 (15h8m3.3458458s) [upload=15670]: Upload finished. err: <nil> Aug 8 00:49:01.779484346 (15h9m1.368829333s) [upload=15683]: Starting upload Aug 8 00:49:01.779541455 (15h9m1.368886442s) [upload=15683]: Done starting upload Aug 8 00:49:01.779591250 (15h9m1.368936238s) [upload=15683]: Uploading segments... Aug 8 00:49:01.779632399 (15h9m1.368977386s) [upload=15683]: Got next segment. Inline: true Aug 8 00:49:01.779637188 (15h9m1.368982185s) [upload=15683]: Waiting for error group managing segments... Aug 8 00:49:01.779673187 (15h9m1.369018174s) [upload=15683]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:49:01.779677375 (15h9m1.369022362s) [upload=15683]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:49:01.779680040 (15h9m1.369025027s) [upload=15683]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:49:02.060683371 (15h9m1.650028378s) [upload=15683]: Done uploading segments... Aug 8 00:49:02.060710513 (15h9m1.6500555s) [upload=15683]: Upload finished. err: <nil> Aug 8 00:50:06.726842474 (15h10m6.316187461s) [upload=15705]: Starting upload Aug 8 00:50:06.726885055 (15h10m6.316230032s) [upload=15705]: Done starting upload Aug 8 00:50:06.726932025 (15h10m6.316277012s) [upload=15705]: Uploading segments... Aug 8 00:50:06.726962704 (15h10m6.316307691s) [upload=15705]: Got next segment. Inline: true Aug 8 00:50:06.726966831 (15h10m6.316311808s) [upload=15705]: Waiting for error group managing segments... Aug 8 00:50:06.726999844 (15h10m6.316344831s) [upload=15705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:50:06.727003823 (15h10m6.31634881s) [upload=15705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:50:06.727006187 (15h10m6.316351164s) [upload=15705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:50:07.002622120 (15h10m6.591967127s) [upload=15705]: Done uploading segments... Aug 8 00:50:07.002639403 (15h10m6.59198438s) [upload=15705]: Upload finished. err: <nil> Aug 8 00:51:18.800365702 (15h11m18.389710689s) [upload=15723]: Starting upload Aug 8 00:51:18.800416448 (15h11m18.389761435s) [upload=15723]: Done starting upload Aug 8 00:51:18.800490300 (15h11m18.389835277s) [upload=15723]: Uploading segments... Aug 8 00:51:18.800624426 (15h11m18.389969413s) [upload=15723]: Got next segment. Inline: true Aug 8 00:51:18.800629807 (15h11m18.389974794s) [upload=15723]: Waiting for error group managing segments... Aug 8 00:51:18.800677268 (15h11m18.390022255s) [upload=15723]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:51:18.800682698 (15h11m18.390027685s) [upload=15723]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:51:18.800685353 (15h11m18.39003034s) [upload=15723]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:51:19.015137684 (15h11m18.604482671s) [upload=15723]: Done uploading segments... Aug 8 00:51:19.015155639 (15h11m18.604500616s) [upload=15723]: Upload finished. err: metaclient: Too Many Requests Duration: 1.881952721s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X.......X...X............X....X.....................] duration: 6.1089439s FAIL (2025-08-08 00:54:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837172821 (14h59m54.426517798s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837385949 (14h59m54.426730936s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837507802 (14h59m54.426852779s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837640145 (14h59m54.426985122s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837819459 (14h59m54.427164446s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837962502 (14h59m54.427307479s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838112790 (14h59m54.427457767s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838238691 (14h59m54.427583678s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838393447 (14h59m54.427738434s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838490402 (14h59m54.427835389s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838660288 (14h59m54.428005275s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838681548 (14h59m54.428026525s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838866312 (14h59m54.428211299s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838893975 (14h59m54.428238962s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.839056986 (14h59m54.428401974s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:55.935384249 (14h59m55.524729246s) [upload=15536]: Next returned nil segment Aug 8 00:39:55.935402815 (14h59m55.524747802s) [upload=15536]: Waiting for error group managing segments... Aug 8 00:39:56.226788010 (14h59m55.816132997s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.265710491 (14h59m55.855055478s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.326363980 (14h59m55.915708967s) [noise=true , address=64.49.13.169:4709 , piece_id=PIPH24KH4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773039652 (14h59m56.362384679s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773213735 (14h59m56.362558732s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.782851711 (14h59m56.372196698s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854376586 (14h59m56.443721573s) [noise=true , address=104.36.82.46:4484 , piece_id=BMHSO2DIXVRRHUZRHWUQ24TXMMTRMVX34SVDUUUMRMKPSUQG3APQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854703231 (14h59m56.444048218s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.914818370 (14h59m56.504163357s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.915843802 (14h59m56.505188849s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.942005621 (14h59m56.531350608s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.981103286 (14h59m56.570448273s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.095564053 (14h59m56.68490904s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216182613 (14h59m56.8055276s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216206869 (14h59m56.805551856s) [seg_pos={0 0}, upload=15536]: Segment reached optimal threshold of 14 pieces. Aug 8 00:39:57.216406632 (14h59m56.805751619s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bN25eEEjxCWKJSr8qj7moDJxArgfsjzPTdBcLdfvoeyxiqPpc): protocol: expected piece hash; context canceled; EOF Aug 8 00:39:57.216460465 (14h59m56.805805452s) [noise=true , address=104.36.82.5:4869 , piece_id=JBCDCG676BBV6U2GAVWDQIWYXKKQGW2DKK6MDY74LNPGP4K3CTQA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection Aug 8 00:39:57.216644467 (14h59m56.805989454s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD): protocol: read tcp 188.166.225.170:50728->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216665567 (14h59m56.806010554s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: read tcp 188.166.225.170:45052->104.36.82.61:4383: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216679534 (14h59m56.806024521s) [seg_pos={0 0}, upload=15536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:39:57.216739699 (14h59m56.806084686s) [upload=15536]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:39:57.216743286 (14h59m56.806088273s) [upload=15536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:39:57.516798834 (14h59m57.106143821s) [upload=15536]: Done uploading segments... Aug 8 00:39:57.516817129 (14h59m57.106162116s) [upload=15536]: Upload finished. err: <nil> Aug 8 00:40:09.579830255 (15h0m9.169175242s) [upload=15541]: Starting upload Aug 8 00:40:09.579877025 (15h0m9.169222012s) [upload=15541]: Done starting upload Aug 8 00:40:09.579941739 (15h0m9.169286726s) [upload=15541]: Uploading segments... Aug 8 00:40:09.580164405 (15h0m9.169509432s) [upload=15541]: Got next segment. Inline: false Aug 8 00:40:09.580219460 (15h0m9.169564477s) [upload=15541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:40:09.580251472 (15h0m9.169596489s) [upload=15541]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:40:09.797368422 (15h0m9.386713409s) [upload=15541]: Next segment err: metaclient: Too Many Requests Aug 8 00:40:09.797398149 (15h0m9.386743136s) [upload=15541]: Done uploading segments... Aug 8 00:40:09.797473614 (15h0m9.386818601s) [upload=15541]: Upload finished. err: metaclient: Too Many Requests Duration: 6.10896429s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.853282456s FAIL (2025-08-08 00:54:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9109253s) [upload=14753]: Starting upload Aug 7 23:54:04.489828037 (14h14m4.079173014s) [upload=14753]: Done starting upload Aug 7 23:54:04.489877612 (14h14m4.079222589s) [upload=14753]: Uploading segments... Aug 7 23:54:04.489903933 (14h14m4.07924892s) [upload=14753]: Got next segment. Inline: true Aug 7 23:54:04.489907469 (14h14m4.079252446s) [upload=14753]: Waiting for error group managing segments... Aug 7 23:54:04.489939651 (14h14m4.079284638s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:04.489943298 (14h14m4.079288285s) [upload=14753]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:04.489945472 (14h14m4.079290459s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:04.778726062 (14h14m4.368071049s) [upload=14753]: Done uploading segments... Aug 7 23:54:04.778744989 (14h14m4.368089976s) [upload=14753]: Upload finished. err: <nil> Aug 7 23:55:14.199719855 (14h15m13.789064892s) [upload=14770]: Starting upload Aug 7 23:55:14.199777645 (14h15m13.789122632s) [upload=14770]: Done starting upload Aug 7 23:55:14.199824415 (14h15m13.789169392s) [upload=14770]: Uploading segments... Aug 7 23:55:14.199852819 (14h15m13.789197806s) [upload=14770]: Got next segment. Inline: true Aug 7 23:55:14.199856436 (14h15m13.789201423s) [upload=14770]: Waiting for error group managing segments... Aug 7 23:55:14.199889419 (14h15m13.789234406s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:14.199893557 (14h15m13.789238544s) [upload=14770]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:14.199895951 (14h15m13.789240938s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:13.542516296 (14h16m13.131861283s) [upload=14770]: Done uploading segments... Aug 7 23:56:13.542536334 (14h16m13.131881321s) [upload=14770]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:14.428677736 (14h16m14.018022723s) [upload=14788]: Starting upload Aug 7 23:56:14.428730978 (14h16m14.018075965s) [upload=14788]: Done starting upload Aug 7 23:56:14.428789289 (14h16m14.018134276s) [upload=14788]: Uploading segments... Aug 7 23:56:14.428819026 (14h16m14.018164013s) [upload=14788]: Got next segment. Inline: true Aug 7 23:56:14.428822873 (14h16m14.01816785s) [upload=14788]: Waiting for error group managing segments... Aug 7 23:56:14.428855215 (14h16m14.018200202s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:14.428858562 (14h16m14.018203549s) [upload=14788]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:14.428860736 (14h16m14.018205723s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:13.781984834 (14h17m13.371329821s) [upload=14788]: Done uploading segments... Aug 7 23:57:13.782003370 (14h17m13.371348357s) [upload=14788]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:29.079359478 (14h17m28.668704466s) [upload=14808]: Starting upload Aug 7 23:57:29.079416147 (14h17m28.668761134s) [upload=14808]: Done starting upload Aug 7 23:57:29.079461845 (14h17m28.668806822s) [upload=14808]: Uploading segments... Aug 7 23:57:29.079488876 (14h17m28.668833863s) [upload=14808]: Got next segment. Inline: true Aug 7 23:57:29.079492263 (14h17m28.66883725s) [upload=14808]: Waiting for error group managing segments... Aug 7 23:57:29.079523392 (14h17m28.668868379s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:29.079527049 (14h17m28.668872036s) [upload=14808]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:29.079529484 (14h17m28.668874461s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:28.439377526 (14h18m28.028722513s) [upload=14808]: Done uploading segments... Aug 7 23:58:28.439395721 (14h18m28.028740708s) [upload=14808]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:29.333669502 (14h18m28.923014489s) [upload=14825]: Starting upload Aug 7 23:58:29.333742581 (14h18m28.923087568s) [upload=14825]: Done starting upload Aug 7 23:58:29.333835740 (14h18m28.923180727s) [upload=14825]: Uploading segments... Aug 7 23:58:29.333878000 (14h18m28.923222987s) [upload=14825]: Got next segment. Inline: true Aug 7 23:58:29.333881998 (14h18m28.923226975s) [upload=14825]: Waiting for error group managing segments... Aug 7 23:58:29.333911455 (14h18m28.923256442s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:29.333917496 (14h18m28.923262473s) [upload=14825]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:29.333919741 (14h18m28.923264728s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:28.692415040 (14h19m28.281760027s) [upload=14825]: Done uploading segments... Aug 7 23:59:28.692444777 (14h19m28.281789764s) [upload=14825]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:29.562470053 (14h19m29.15181504s) [upload=14840]: Starting upload Aug 7 23:59:29.562538985 (14h19m29.151883972s) [upload=14840]: Done starting upload Aug 7 23:59:29.562614629 (14h19m29.151959616s) [upload=14840]: Uploading segments... Aug 7 23:59:29.562650589 (14h19m29.151995566s) [upload=14840]: Got next segment. Inline: true Aug 7 23:59:29.562655498 (14h19m29.152000485s) [upload=14840]: Waiting for error group managing segments... Aug 7 23:59:29.562686507 (14h19m29.152031494s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:29.562690264 (14h19m29.152035241s) [upload=14840]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:29.562692428 (14h19m29.152037405s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:29.822497550 (14h19m29.411842537s) [upload=14840]: Done uploading segments... Aug 7 23:59:29.822514693 (14h19m29.41185968s) [upload=14840]: Upload finished. err: <nil> Aug 8 00:00:35.820554363 (14h20m35.40989935s) [upload=14856]: Starting upload Aug 8 00:00:35.820610611 (14h20m35.409955598s) [upload=14856]: Done starting upload Aug 8 00:00:35.820669503 (14h20m35.4100145s) [upload=14856]: Uploading segments... Aug 8 00:00:35.820742333 (14h20m35.41008732s) [upload=14856]: Got next segment. Inline: true Aug 8 00:00:35.820747282 (14h20m35.410092269s) [upload=14856]: Waiting for error group managing segments... Aug 8 00:00:35.820788371 (14h20m35.410133358s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.820793050 (14h20m35.410138037s) [upload=14856]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.820795485 (14h20m35.410140472s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:36.084321816 (14h20m35.673666803s) [upload=14856]: Done uploading segments... Aug 8 00:00:36.084338618 (14h20m35.673683615s) [upload=14856]: Upload finished. err: <nil> Aug 8 00:01:14.496466774 (14h21m14.085811781s) [upload=14868]: Starting upload Aug 8 00:01:14.496556174 (14h21m14.085901162s) [upload=14868]: Done starting upload Aug 8 00:01:14.496609206 (14h21m14.085954183s) [upload=14868]: Uploading segments... Aug 8 00:01:14.496639985 (14h21m14.085984972s) [upload=14868]: Got next segment. Inline: true Aug 8 00:01:14.496643592 (14h21m14.085988579s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:01:14.496678638 (14h21m14.086023625s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:14.496682386 (14h21m14.086027373s) [upload=14868]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:14.496684681 (14h21m14.086029658s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:14.760681029 (14h21m14.350026016s) [upload=14868]: Done uploading segments... Aug 8 00:01:14.760702580 (14h21m14.350047567s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:01:50.577049881 (14h21m50.166394858s) [upload=14878]: Starting upload Aug 8 00:01:50.577100538 (14h21m50.166445525s) [upload=14878]: Done starting upload Aug 8 00:01:50.577180091 (14h21m50.166525078s) [upload=14878]: Uploading segments... Aug 8 00:01:50.577228012 (14h21m50.166573s) [upload=14878]: Got next segment. Inline: true Aug 8 00:01:50.577231469 (14h21m50.166576456s) [upload=14878]: Waiting for error group managing segments... Aug 8 00:01:50.577270633 (14h21m50.166615621s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:50.577274721 (14h21m50.166619708s) [upload=14878]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:50.577276906 (14h21m50.166621893s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:50.863237339 (14h21m50.452582326s) [upload=14878]: Done uploading segments... Aug 8 00:01:50.863254973 (14h21m50.452599961s) [upload=14878]: Upload finished. err: <nil> Aug 8 00:02:41.474648957 (14h22m41.063993944s) [upload=14899]: Starting upload Aug 8 00:02:41.474716297 (14h22m41.064061274s) [upload=14899]: Done starting upload Aug 8 00:02:41.474771743 (14h22m41.06411673s) [upload=14899]: Uploading segments... Aug 8 00:02:41.474801099 (14h22m41.064146086s) [upload=14899]: Got next segment. Inline: true Aug 8 00:02:41.474804966 (14h22m41.064149953s) [upload=14899]: Waiting for error group managing segments... Aug 8 00:02:41.474841336 (14h22m41.064186323s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:41.474845214 (14h22m41.064190201s) [upload=14899]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:41.474847558 (14h22m41.064192535s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:03:40.835731182 (14h23m40.425076189s) [upload=14899]: Done uploading segments... Aug 8 00:03:40.835778212 (14h23m40.425123199s) [upload=14899]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.853317442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X.......X...X............X....X....................] duration: 6.437276101s FAIL (2025-08-08 00:53:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837172821 (14h59m54.426517798s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837385949 (14h59m54.426730936s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837507802 (14h59m54.426852779s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837640145 (14h59m54.426985122s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837819459 (14h59m54.427164446s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837962502 (14h59m54.427307479s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838112790 (14h59m54.427457767s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838238691 (14h59m54.427583678s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838393447 (14h59m54.427738434s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838490402 (14h59m54.427835389s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838660288 (14h59m54.428005275s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838681548 (14h59m54.428026525s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838866312 (14h59m54.428211299s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838893975 (14h59m54.428238962s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.839056986 (14h59m54.428401974s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:55.935384249 (14h59m55.524729246s) [upload=15536]: Next returned nil segment Aug 8 00:39:55.935402815 (14h59m55.524747802s) [upload=15536]: Waiting for error group managing segments... Aug 8 00:39:56.226788010 (14h59m55.816132997s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.265710491 (14h59m55.855055478s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.326363980 (14h59m55.915708967s) [noise=true , address=64.49.13.169:4709 , piece_id=PIPH24KH4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773039652 (14h59m56.362384679s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773213735 (14h59m56.362558732s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.782851711 (14h59m56.372196698s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854376586 (14h59m56.443721573s) [noise=true , address=104.36.82.46:4484 , piece_id=BMHSO2DIXVRRHUZRHWUQ24TXMMTRMVX34SVDUUUMRMKPSUQG3APQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854703231 (14h59m56.444048218s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.914818370 (14h59m56.504163357s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.915843802 (14h59m56.505188849s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.942005621 (14h59m56.531350608s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.981103286 (14h59m56.570448273s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.095564053 (14h59m56.68490904s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216182613 (14h59m56.8055276s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216206869 (14h59m56.805551856s) [seg_pos={0 0}, upload=15536]: Segment reached optimal threshold of 14 pieces. Aug 8 00:39:57.216406632 (14h59m56.805751619s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bN25eEEjxCWKJSr8qj7moDJxArgfsjzPTdBcLdfvoeyxiqPpc): protocol: expected piece hash; context canceled; EOF Aug 8 00:39:57.216460465 (14h59m56.805805452s) [noise=true , address=104.36.82.5:4869 , piece_id=JBCDCG676BBV6U2GAVWDQIWYXKKQGW2DKK6MDY74LNPGP4K3CTQA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection Aug 8 00:39:57.216644467 (14h59m56.805989454s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD): protocol: read tcp 188.166.225.170:50728->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216665567 (14h59m56.806010554s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: read tcp 188.166.225.170:45052->104.36.82.61:4383: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216679534 (14h59m56.806024521s) [seg_pos={0 0}, upload=15536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:39:57.216739699 (14h59m56.806084686s) [upload=15536]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:39:57.216743286 (14h59m56.806088273s) [upload=15536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:39:57.516798834 (14h59m57.106143821s) [upload=15536]: Done uploading segments... Aug 8 00:39:57.516817129 (14h59m57.106162116s) [upload=15536]: Upload finished. err: <nil> Aug 8 00:40:09.579830255 (15h0m9.169175242s) [upload=15541]: Starting upload Aug 8 00:40:09.579877025 (15h0m9.169222012s) [upload=15541]: Done starting upload Aug 8 00:40:09.579941739 (15h0m9.169286726s) [upload=15541]: Uploading segments... Aug 8 00:40:09.580164405 (15h0m9.169509432s) [upload=15541]: Got next segment. Inline: false Aug 8 00:40:09.580219460 (15h0m9.169564477s) [upload=15541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:40:09.580251472 (15h0m9.169596489s) [upload=15541]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:40:09.797368422 (15h0m9.386713409s) [upload=15541]: Next segment err: metaclient: Too Many Requests Aug 8 00:40:09.797398149 (15h0m9.386743136s) [upload=15541]: Done uploading segments... Aug 8 00:40:09.797473614 (15h0m9.386818601s) [upload=15541]: Upload finished. err: metaclient: Too Many Requests Duration: 6.437304904s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............X.........................................X...] duration: 1.984581591s FAIL (2025-08-08 00:53:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== stItem_ObjectCommit Aug 8 00:42:17.713078039 (15h2m17.302423046s) [upload=15574]: Done uploading segments... Aug 8 00:42:17.713100702 (15h2m17.30244574s) [upload=15574]: Upload finished. err: <nil> Aug 8 00:42:36.033866449 (15h2m35.623211436s) [upload=15582]: Starting upload Aug 8 00:42:36.033918057 (15h2m35.623263044s) [upload=15582]: Done starting upload Aug 8 00:42:36.033968163 (15h2m35.623313141s) [upload=15582]: Uploading segments... Aug 8 00:42:36.033998922 (15h2m35.623343909s) [upload=15582]: Got next segment. Inline: true Aug 8 00:42:36.034002399 (15h2m35.623347386s) [upload=15582]: Waiting for error group managing segments... Aug 8 00:42:36.034039881 (15h2m35.623384868s) [upload=15582]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:42:36.034043698 (15h2m35.623388685s) [upload=15582]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:42:36.034046063 (15h2m35.62339105s) [upload=15582]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:42:36.326408369 (15h2m35.915753356s) [upload=15582]: Done uploading segments... Aug 8 00:42:36.326427806 (15h2m35.915772823s) [upload=15582]: Upload finished. err: <nil> Aug 8 00:43:50.785976974 (15h3m50.375321961s) [upload=15600]: Starting upload Aug 8 00:43:50.786021608 (15h3m50.375366585s) [upload=15600]: Done starting upload Aug 8 00:43:50.786068328 (15h3m50.375413315s) [upload=15600]: Uploading segments... Aug 8 00:43:50.786096292 (15h3m50.375441269s) [upload=15600]: Got next segment. Inline: true Aug 8 00:43:50.786099928 (15h3m50.375444915s) [upload=15600]: Waiting for error group managing segments... Aug 8 00:43:50.786133022 (15h3m50.375478009s) [upload=15600]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:43:50.786136840 (15h3m50.375481817s) [upload=15600]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:43:50.786139374 (15h3m50.375484351s) [upload=15600]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:43:51.071276196 (15h3m50.660621183s) [upload=15600]: Done uploading segments... Aug 8 00:43:51.071294140 (15h3m50.660639127s) [upload=15600]: Upload finished. err: <nil> Aug 8 00:44:38.932060864 (15h4m38.521405851s) [upload=15614]: Starting upload Aug 8 00:44:38.932105570 (15h4m38.521450547s) [upload=15614]: Done starting upload Aug 8 00:44:38.932150475 (15h4m38.521495462s) [upload=15614]: Uploading segments... Aug 8 00:44:38.932180272 (15h4m38.521525249s) [upload=15614]: Got next segment. Inline: true Aug 8 00:44:38.932183829 (15h4m38.521528816s) [upload=15614]: Waiting for error group managing segments... Aug 8 00:44:38.932214318 (15h4m38.521559305s) [upload=15614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:44:38.932218205 (15h4m38.521563192s) [upload=15614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:44:38.932220419 (15h4m38.521565396s) [upload=15614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:44:39.257010779 (15h4m38.846355757s) [upload=15614]: Done uploading segments... Aug 8 00:44:39.257027903 (15h4m38.84637289s) [upload=15614]: Upload finished. err: <nil> Aug 8 00:44:55.671929119 (15h4m55.261274106s) [upload=15619]: Starting upload Aug 8 00:44:55.671974105 (15h4m55.261319092s) [upload=15619]: Done starting upload Aug 8 00:44:55.672033319 (15h4m55.261378296s) [upload=15619]: Uploading segments... Aug 8 00:44:55.672073296 (15h4m55.261418273s) [upload=15619]: Got next segment. Inline: true Aug 8 00:44:55.672077514 (15h4m55.261422501s) [upload=15619]: Waiting for error group managing segments... Aug 8 00:44:55.672111548 (15h4m55.261456535s) [upload=15619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:44:55.672115085 (15h4m55.261460072s) [upload=15619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:44:55.672117520 (15h4m55.261462507s) [upload=15619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:44:55.946154002 (15h4m55.535498989s) [upload=15619]: Done uploading segments... Aug 8 00:44:55.946180583 (15h4m55.53552557s) [upload=15619]: Upload finished. err: <nil> Aug 8 00:45:51.335687032 (15h5m50.925032019s) [upload=15634]: Starting upload Aug 8 00:45:51.335741947 (15h5m50.925086934s) [upload=15634]: Done starting upload Aug 8 00:45:51.335838182 (15h5m50.925183169s) [upload=15634]: Uploading segments... Aug 8 00:45:51.335870213 (15h5m50.9252152s) [upload=15634]: Got next segment. Inline: true Aug 8 00:45:51.335873930 (15h5m50.925218917s) [upload=15634]: Waiting for error group managing segments... Aug 8 00:45:51.335917033 (15h5m50.92526202s) [upload=15634]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:45:51.335921060 (15h5m50.925266047s) [upload=15634]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:45:51.335923345 (15h5m50.925268332s) [upload=15634]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:45:51.622624488 (15h5m51.211969475s) [upload=15634]: Done uploading segments... Aug 8 00:45:51.622650428 (15h5m51.211995415s) [upload=15634]: Upload finished. err: <nil> Aug 8 00:46:56.158030766 (15h6m55.747375753s) [upload=15655]: Starting upload Aug 8 00:46:56.158075261 (15h6m55.747420248s) [upload=15655]: Done starting upload Aug 8 00:46:56.158127792 (15h6m55.747472779s) [upload=15655]: Uploading segments... Aug 8 00:46:56.158159252 (15h6m55.747504229s) [upload=15655]: Got next segment. Inline: true Aug 8 00:46:56.158162568 (15h6m55.747507555s) [upload=15655]: Waiting for error group managing segments... Aug 8 00:46:56.158193087 (15h6m55.747538074s) [upload=15655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:46:56.158196653 (15h6m55.74754164s) [upload=15655]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:46:56.158199098 (15h6m55.747544075s) [upload=15655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:46:56.421913161 (15h6m56.011258168s) [upload=15655]: Done uploading segments... Aug 8 00:46:56.421936716 (15h6m56.011281703s) [upload=15655]: Upload finished. err: <nil> Aug 8 00:48:03.458104986 (15h8m3.047449973s) [upload=15670]: Starting upload Aug 8 00:48:03.458166774 (15h8m3.047511761s) [upload=15670]: Done starting upload Aug 8 00:48:03.458226910 (15h8m3.047571887s) [upload=15670]: Uploading segments... Aug 8 00:48:03.458331268 (15h8m3.047676255s) [upload=15670]: Got next segment. Inline: true Aug 8 00:48:03.458336880 (15h8m3.047681857s) [upload=15670]: Waiting for error group managing segments... Aug 8 00:48:03.458383799 (15h8m3.047728786s) [upload=15670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:48:03.458388839 (15h8m3.047733827s) [upload=15670]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:48:03.458391154 (15h8m3.047736141s) [upload=15670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:48:03.756480234 (15h8m3.345825221s) [upload=15670]: Done uploading segments... Aug 8 00:48:03.756500813 (15h8m3.3458458s) [upload=15670]: Upload finished. err: <nil> Aug 8 00:49:01.779484346 (15h9m1.368829333s) [upload=15683]: Starting upload Aug 8 00:49:01.779541455 (15h9m1.368886442s) [upload=15683]: Done starting upload Aug 8 00:49:01.779591250 (15h9m1.368936238s) [upload=15683]: Uploading segments... Aug 8 00:49:01.779632399 (15h9m1.368977386s) [upload=15683]: Got next segment. Inline: true Aug 8 00:49:01.779637188 (15h9m1.368982185s) [upload=15683]: Waiting for error group managing segments... Aug 8 00:49:01.779673187 (15h9m1.369018174s) [upload=15683]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:49:01.779677375 (15h9m1.369022362s) [upload=15683]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:49:01.779680040 (15h9m1.369025027s) [upload=15683]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:49:02.060683371 (15h9m1.650028378s) [upload=15683]: Done uploading segments... Aug 8 00:49:02.060710513 (15h9m1.6500555s) [upload=15683]: Upload finished. err: <nil> Aug 8 00:50:06.726842474 (15h10m6.316187461s) [upload=15705]: Starting upload Aug 8 00:50:06.726885055 (15h10m6.316230032s) [upload=15705]: Done starting upload Aug 8 00:50:06.726932025 (15h10m6.316277012s) [upload=15705]: Uploading segments... Aug 8 00:50:06.726962704 (15h10m6.316307691s) [upload=15705]: Got next segment. Inline: true Aug 8 00:50:06.726966831 (15h10m6.316311808s) [upload=15705]: Waiting for error group managing segments... Aug 8 00:50:06.726999844 (15h10m6.316344831s) [upload=15705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:50:06.727003823 (15h10m6.31634881s) [upload=15705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:50:06.727006187 (15h10m6.316351164s) [upload=15705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:50:07.002622120 (15h10m6.591967127s) [upload=15705]: Done uploading segments... Aug 8 00:50:07.002639403 (15h10m6.59198438s) [upload=15705]: Upload finished. err: <nil> Aug 8 00:51:18.800365702 (15h11m18.389710689s) [upload=15723]: Starting upload Aug 8 00:51:18.800416448 (15h11m18.389761435s) [upload=15723]: Done starting upload Aug 8 00:51:18.800490300 (15h11m18.389835277s) [upload=15723]: Uploading segments... Aug 8 00:51:18.800624426 (15h11m18.389969413s) [upload=15723]: Got next segment. Inline: true Aug 8 00:51:18.800629807 (15h11m18.389974794s) [upload=15723]: Waiting for error group managing segments... Aug 8 00:51:18.800677268 (15h11m18.390022255s) [upload=15723]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:51:18.800682698 (15h11m18.390027685s) [upload=15723]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:51:18.800685353 (15h11m18.39003034s) [upload=15723]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:51:19.015137684 (15h11m18.604482671s) [upload=15723]: Done uploading segments... Aug 8 00:51:19.015155639 (15h11m18.604500616s) [upload=15723]: Upload finished. err: metaclient: Too Many Requests Duration: 1.98461862s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.916911799s FAIL (2025-08-08 00:53:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9109253s) [upload=14753]: Starting upload Aug 7 23:54:04.489828037 (14h14m4.079173014s) [upload=14753]: Done starting upload Aug 7 23:54:04.489877612 (14h14m4.079222589s) [upload=14753]: Uploading segments... Aug 7 23:54:04.489903933 (14h14m4.07924892s) [upload=14753]: Got next segment. Inline: true Aug 7 23:54:04.489907469 (14h14m4.079252446s) [upload=14753]: Waiting for error group managing segments... Aug 7 23:54:04.489939651 (14h14m4.079284638s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:04.489943298 (14h14m4.079288285s) [upload=14753]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:04.489945472 (14h14m4.079290459s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:04.778726062 (14h14m4.368071049s) [upload=14753]: Done uploading segments... Aug 7 23:54:04.778744989 (14h14m4.368089976s) [upload=14753]: Upload finished. err: <nil> Aug 7 23:55:14.199719855 (14h15m13.789064892s) [upload=14770]: Starting upload Aug 7 23:55:14.199777645 (14h15m13.789122632s) [upload=14770]: Done starting upload Aug 7 23:55:14.199824415 (14h15m13.789169392s) [upload=14770]: Uploading segments... Aug 7 23:55:14.199852819 (14h15m13.789197806s) [upload=14770]: Got next segment. Inline: true Aug 7 23:55:14.199856436 (14h15m13.789201423s) [upload=14770]: Waiting for error group managing segments... Aug 7 23:55:14.199889419 (14h15m13.789234406s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:14.199893557 (14h15m13.789238544s) [upload=14770]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:14.199895951 (14h15m13.789240938s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:13.542516296 (14h16m13.131861283s) [upload=14770]: Done uploading segments... Aug 7 23:56:13.542536334 (14h16m13.131881321s) [upload=14770]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:14.428677736 (14h16m14.018022723s) [upload=14788]: Starting upload Aug 7 23:56:14.428730978 (14h16m14.018075965s) [upload=14788]: Done starting upload Aug 7 23:56:14.428789289 (14h16m14.018134276s) [upload=14788]: Uploading segments... Aug 7 23:56:14.428819026 (14h16m14.018164013s) [upload=14788]: Got next segment. Inline: true Aug 7 23:56:14.428822873 (14h16m14.01816785s) [upload=14788]: Waiting for error group managing segments... Aug 7 23:56:14.428855215 (14h16m14.018200202s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:14.428858562 (14h16m14.018203549s) [upload=14788]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:14.428860736 (14h16m14.018205723s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:13.781984834 (14h17m13.371329821s) [upload=14788]: Done uploading segments... Aug 7 23:57:13.782003370 (14h17m13.371348357s) [upload=14788]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:29.079359478 (14h17m28.668704466s) [upload=14808]: Starting upload Aug 7 23:57:29.079416147 (14h17m28.668761134s) [upload=14808]: Done starting upload Aug 7 23:57:29.079461845 (14h17m28.668806822s) [upload=14808]: Uploading segments... Aug 7 23:57:29.079488876 (14h17m28.668833863s) [upload=14808]: Got next segment. Inline: true Aug 7 23:57:29.079492263 (14h17m28.66883725s) [upload=14808]: Waiting for error group managing segments... Aug 7 23:57:29.079523392 (14h17m28.668868379s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:29.079527049 (14h17m28.668872036s) [upload=14808]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:29.079529484 (14h17m28.668874461s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:28.439377526 (14h18m28.028722513s) [upload=14808]: Done uploading segments... Aug 7 23:58:28.439395721 (14h18m28.028740708s) [upload=14808]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:29.333669502 (14h18m28.923014489s) [upload=14825]: Starting upload Aug 7 23:58:29.333742581 (14h18m28.923087568s) [upload=14825]: Done starting upload Aug 7 23:58:29.333835740 (14h18m28.923180727s) [upload=14825]: Uploading segments... Aug 7 23:58:29.333878000 (14h18m28.923222987s) [upload=14825]: Got next segment. Inline: true Aug 7 23:58:29.333881998 (14h18m28.923226975s) [upload=14825]: Waiting for error group managing segments... Aug 7 23:58:29.333911455 (14h18m28.923256442s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:29.333917496 (14h18m28.923262473s) [upload=14825]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:29.333919741 (14h18m28.923264728s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:28.692415040 (14h19m28.281760027s) [upload=14825]: Done uploading segments... Aug 7 23:59:28.692444777 (14h19m28.281789764s) [upload=14825]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:29.562470053 (14h19m29.15181504s) [upload=14840]: Starting upload Aug 7 23:59:29.562538985 (14h19m29.151883972s) [upload=14840]: Done starting upload Aug 7 23:59:29.562614629 (14h19m29.151959616s) [upload=14840]: Uploading segments... Aug 7 23:59:29.562650589 (14h19m29.151995566s) [upload=14840]: Got next segment. Inline: true Aug 7 23:59:29.562655498 (14h19m29.152000485s) [upload=14840]: Waiting for error group managing segments... Aug 7 23:59:29.562686507 (14h19m29.152031494s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:29.562690264 (14h19m29.152035241s) [upload=14840]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:29.562692428 (14h19m29.152037405s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:29.822497550 (14h19m29.411842537s) [upload=14840]: Done uploading segments... Aug 7 23:59:29.822514693 (14h19m29.41185968s) [upload=14840]: Upload finished. err: <nil> Aug 8 00:00:35.820554363 (14h20m35.40989935s) [upload=14856]: Starting upload Aug 8 00:00:35.820610611 (14h20m35.409955598s) [upload=14856]: Done starting upload Aug 8 00:00:35.820669503 (14h20m35.4100145s) [upload=14856]: Uploading segments... Aug 8 00:00:35.820742333 (14h20m35.41008732s) [upload=14856]: Got next segment. Inline: true Aug 8 00:00:35.820747282 (14h20m35.410092269s) [upload=14856]: Waiting for error group managing segments... Aug 8 00:00:35.820788371 (14h20m35.410133358s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.820793050 (14h20m35.410138037s) [upload=14856]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.820795485 (14h20m35.410140472s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:36.084321816 (14h20m35.673666803s) [upload=14856]: Done uploading segments... Aug 8 00:00:36.084338618 (14h20m35.673683615s) [upload=14856]: Upload finished. err: <nil> Aug 8 00:01:14.496466774 (14h21m14.085811781s) [upload=14868]: Starting upload Aug 8 00:01:14.496556174 (14h21m14.085901162s) [upload=14868]: Done starting upload Aug 8 00:01:14.496609206 (14h21m14.085954183s) [upload=14868]: Uploading segments... Aug 8 00:01:14.496639985 (14h21m14.085984972s) [upload=14868]: Got next segment. Inline: true Aug 8 00:01:14.496643592 (14h21m14.085988579s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:01:14.496678638 (14h21m14.086023625s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:14.496682386 (14h21m14.086027373s) [upload=14868]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:14.496684681 (14h21m14.086029658s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:14.760681029 (14h21m14.350026016s) [upload=14868]: Done uploading segments... Aug 8 00:01:14.760702580 (14h21m14.350047567s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:01:50.577049881 (14h21m50.166394858s) [upload=14878]: Starting upload Aug 8 00:01:50.577100538 (14h21m50.166445525s) [upload=14878]: Done starting upload Aug 8 00:01:50.577180091 (14h21m50.166525078s) [upload=14878]: Uploading segments... Aug 8 00:01:50.577228012 (14h21m50.166573s) [upload=14878]: Got next segment. Inline: true Aug 8 00:01:50.577231469 (14h21m50.166576456s) [upload=14878]: Waiting for error group managing segments... Aug 8 00:01:50.577270633 (14h21m50.166615621s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:50.577274721 (14h21m50.166619708s) [upload=14878]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:50.577276906 (14h21m50.166621893s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:50.863237339 (14h21m50.452582326s) [upload=14878]: Done uploading segments... Aug 8 00:01:50.863254973 (14h21m50.452599961s) [upload=14878]: Upload finished. err: <nil> Aug 8 00:02:41.474648957 (14h22m41.063993944s) [upload=14899]: Starting upload Aug 8 00:02:41.474716297 (14h22m41.064061274s) [upload=14899]: Done starting upload Aug 8 00:02:41.474771743 (14h22m41.06411673s) [upload=14899]: Uploading segments... Aug 8 00:02:41.474801099 (14h22m41.064146086s) [upload=14899]: Got next segment. Inline: true Aug 8 00:02:41.474804966 (14h22m41.064149953s) [upload=14899]: Waiting for error group managing segments... Aug 8 00:02:41.474841336 (14h22m41.064186323s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:41.474845214 (14h22m41.064190201s) [upload=14899]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:41.474847558 (14h22m41.064192535s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:03:40.835731182 (14h23m40.425076189s) [upload=14899]: Done uploading segments... Aug 8 00:03:40.835778212 (14h23m40.425123199s) [upload=14899]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.916939491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............X.........................................X..] duration: 1.945458836s FAIL (2025-08-08 00:52:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== stItem_ObjectCommit Aug 8 00:42:17.713078039 (15h2m17.302423046s) [upload=15574]: Done uploading segments... Aug 8 00:42:17.713100702 (15h2m17.30244574s) [upload=15574]: Upload finished. err: <nil> Aug 8 00:42:36.033866449 (15h2m35.623211436s) [upload=15582]: Starting upload Aug 8 00:42:36.033918057 (15h2m35.623263044s) [upload=15582]: Done starting upload Aug 8 00:42:36.033968163 (15h2m35.623313141s) [upload=15582]: Uploading segments... Aug 8 00:42:36.033998922 (15h2m35.623343909s) [upload=15582]: Got next segment. Inline: true Aug 8 00:42:36.034002399 (15h2m35.623347386s) [upload=15582]: Waiting for error group managing segments... Aug 8 00:42:36.034039881 (15h2m35.623384868s) [upload=15582]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:42:36.034043698 (15h2m35.623388685s) [upload=15582]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:42:36.034046063 (15h2m35.62339105s) [upload=15582]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:42:36.326408369 (15h2m35.915753356s) [upload=15582]: Done uploading segments... Aug 8 00:42:36.326427806 (15h2m35.915772823s) [upload=15582]: Upload finished. err: <nil> Aug 8 00:43:50.785976974 (15h3m50.375321961s) [upload=15600]: Starting upload Aug 8 00:43:50.786021608 (15h3m50.375366585s) [upload=15600]: Done starting upload Aug 8 00:43:50.786068328 (15h3m50.375413315s) [upload=15600]: Uploading segments... Aug 8 00:43:50.786096292 (15h3m50.375441269s) [upload=15600]: Got next segment. Inline: true Aug 8 00:43:50.786099928 (15h3m50.375444915s) [upload=15600]: Waiting for error group managing segments... Aug 8 00:43:50.786133022 (15h3m50.375478009s) [upload=15600]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:43:50.786136840 (15h3m50.375481817s) [upload=15600]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:43:50.786139374 (15h3m50.375484351s) [upload=15600]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:43:51.071276196 (15h3m50.660621183s) [upload=15600]: Done uploading segments... Aug 8 00:43:51.071294140 (15h3m50.660639127s) [upload=15600]: Upload finished. err: <nil> Aug 8 00:44:38.932060864 (15h4m38.521405851s) [upload=15614]: Starting upload Aug 8 00:44:38.932105570 (15h4m38.521450547s) [upload=15614]: Done starting upload Aug 8 00:44:38.932150475 (15h4m38.521495462s) [upload=15614]: Uploading segments... Aug 8 00:44:38.932180272 (15h4m38.521525249s) [upload=15614]: Got next segment. Inline: true Aug 8 00:44:38.932183829 (15h4m38.521528816s) [upload=15614]: Waiting for error group managing segments... Aug 8 00:44:38.932214318 (15h4m38.521559305s) [upload=15614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:44:38.932218205 (15h4m38.521563192s) [upload=15614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:44:38.932220419 (15h4m38.521565396s) [upload=15614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:44:39.257010779 (15h4m38.846355757s) [upload=15614]: Done uploading segments... Aug 8 00:44:39.257027903 (15h4m38.84637289s) [upload=15614]: Upload finished. err: <nil> Aug 8 00:44:55.671929119 (15h4m55.261274106s) [upload=15619]: Starting upload Aug 8 00:44:55.671974105 (15h4m55.261319092s) [upload=15619]: Done starting upload Aug 8 00:44:55.672033319 (15h4m55.261378296s) [upload=15619]: Uploading segments... Aug 8 00:44:55.672073296 (15h4m55.261418273s) [upload=15619]: Got next segment. Inline: true Aug 8 00:44:55.672077514 (15h4m55.261422501s) [upload=15619]: Waiting for error group managing segments... Aug 8 00:44:55.672111548 (15h4m55.261456535s) [upload=15619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:44:55.672115085 (15h4m55.261460072s) [upload=15619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:44:55.672117520 (15h4m55.261462507s) [upload=15619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:44:55.946154002 (15h4m55.535498989s) [upload=15619]: Done uploading segments... Aug 8 00:44:55.946180583 (15h4m55.53552557s) [upload=15619]: Upload finished. err: <nil> Aug 8 00:45:51.335687032 (15h5m50.925032019s) [upload=15634]: Starting upload Aug 8 00:45:51.335741947 (15h5m50.925086934s) [upload=15634]: Done starting upload Aug 8 00:45:51.335838182 (15h5m50.925183169s) [upload=15634]: Uploading segments... Aug 8 00:45:51.335870213 (15h5m50.9252152s) [upload=15634]: Got next segment. Inline: true Aug 8 00:45:51.335873930 (15h5m50.925218917s) [upload=15634]: Waiting for error group managing segments... Aug 8 00:45:51.335917033 (15h5m50.92526202s) [upload=15634]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:45:51.335921060 (15h5m50.925266047s) [upload=15634]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:45:51.335923345 (15h5m50.925268332s) [upload=15634]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:45:51.622624488 (15h5m51.211969475s) [upload=15634]: Done uploading segments... Aug 8 00:45:51.622650428 (15h5m51.211995415s) [upload=15634]: Upload finished. err: <nil> Aug 8 00:46:56.158030766 (15h6m55.747375753s) [upload=15655]: Starting upload Aug 8 00:46:56.158075261 (15h6m55.747420248s) [upload=15655]: Done starting upload Aug 8 00:46:56.158127792 (15h6m55.747472779s) [upload=15655]: Uploading segments... Aug 8 00:46:56.158159252 (15h6m55.747504229s) [upload=15655]: Got next segment. Inline: true Aug 8 00:46:56.158162568 (15h6m55.747507555s) [upload=15655]: Waiting for error group managing segments... Aug 8 00:46:56.158193087 (15h6m55.747538074s) [upload=15655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:46:56.158196653 (15h6m55.74754164s) [upload=15655]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:46:56.158199098 (15h6m55.747544075s) [upload=15655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:46:56.421913161 (15h6m56.011258168s) [upload=15655]: Done uploading segments... Aug 8 00:46:56.421936716 (15h6m56.011281703s) [upload=15655]: Upload finished. err: <nil> Aug 8 00:48:03.458104986 (15h8m3.047449973s) [upload=15670]: Starting upload Aug 8 00:48:03.458166774 (15h8m3.047511761s) [upload=15670]: Done starting upload Aug 8 00:48:03.458226910 (15h8m3.047571887s) [upload=15670]: Uploading segments... Aug 8 00:48:03.458331268 (15h8m3.047676255s) [upload=15670]: Got next segment. Inline: true Aug 8 00:48:03.458336880 (15h8m3.047681857s) [upload=15670]: Waiting for error group managing segments... Aug 8 00:48:03.458383799 (15h8m3.047728786s) [upload=15670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:48:03.458388839 (15h8m3.047733827s) [upload=15670]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:48:03.458391154 (15h8m3.047736141s) [upload=15670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:48:03.756480234 (15h8m3.345825221s) [upload=15670]: Done uploading segments... Aug 8 00:48:03.756500813 (15h8m3.3458458s) [upload=15670]: Upload finished. err: <nil> Aug 8 00:49:01.779484346 (15h9m1.368829333s) [upload=15683]: Starting upload Aug 8 00:49:01.779541455 (15h9m1.368886442s) [upload=15683]: Done starting upload Aug 8 00:49:01.779591250 (15h9m1.368936238s) [upload=15683]: Uploading segments... Aug 8 00:49:01.779632399 (15h9m1.368977386s) [upload=15683]: Got next segment. Inline: true Aug 8 00:49:01.779637188 (15h9m1.368982185s) [upload=15683]: Waiting for error group managing segments... Aug 8 00:49:01.779673187 (15h9m1.369018174s) [upload=15683]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:49:01.779677375 (15h9m1.369022362s) [upload=15683]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:49:01.779680040 (15h9m1.369025027s) [upload=15683]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:49:02.060683371 (15h9m1.650028378s) [upload=15683]: Done uploading segments... Aug 8 00:49:02.060710513 (15h9m1.6500555s) [upload=15683]: Upload finished. err: <nil> Aug 8 00:50:06.726842474 (15h10m6.316187461s) [upload=15705]: Starting upload Aug 8 00:50:06.726885055 (15h10m6.316230032s) [upload=15705]: Done starting upload Aug 8 00:50:06.726932025 (15h10m6.316277012s) [upload=15705]: Uploading segments... Aug 8 00:50:06.726962704 (15h10m6.316307691s) [upload=15705]: Got next segment. Inline: true Aug 8 00:50:06.726966831 (15h10m6.316311808s) [upload=15705]: Waiting for error group managing segments... Aug 8 00:50:06.726999844 (15h10m6.316344831s) [upload=15705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:50:06.727003823 (15h10m6.31634881s) [upload=15705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:50:06.727006187 (15h10m6.316351164s) [upload=15705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:50:07.002622120 (15h10m6.591967127s) [upload=15705]: Done uploading segments... Aug 8 00:50:07.002639403 (15h10m6.59198438s) [upload=15705]: Upload finished. err: <nil> Aug 8 00:51:18.800365702 (15h11m18.389710689s) [upload=15723]: Starting upload Aug 8 00:51:18.800416448 (15h11m18.389761435s) [upload=15723]: Done starting upload Aug 8 00:51:18.800490300 (15h11m18.389835277s) [upload=15723]: Uploading segments... Aug 8 00:51:18.800624426 (15h11m18.389969413s) [upload=15723]: Got next segment. Inline: true Aug 8 00:51:18.800629807 (15h11m18.389974794s) [upload=15723]: Waiting for error group managing segments... Aug 8 00:51:18.800677268 (15h11m18.390022255s) [upload=15723]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:51:18.800682698 (15h11m18.390027685s) [upload=15723]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:51:18.800685353 (15h11m18.39003034s) [upload=15723]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:51:19.015137684 (15h11m18.604482671s) [upload=15723]: Done uploading segments... Aug 8 00:51:19.015155639 (15h11m18.604500616s) [upload=15723]: Upload finished. err: metaclient: Too Many Requests Duration: 1.94549219s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X.......X...X............X....X...................] duration: 6.219319252s FAIL (2025-08-08 00:52:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837172821 (14h59m54.426517798s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837385949 (14h59m54.426730936s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837507802 (14h59m54.426852779s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837640145 (14h59m54.426985122s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837819459 (14h59m54.427164446s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837962502 (14h59m54.427307479s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838112790 (14h59m54.427457767s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838238691 (14h59m54.427583678s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838393447 (14h59m54.427738434s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838490402 (14h59m54.427835389s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838660288 (14h59m54.428005275s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838681548 (14h59m54.428026525s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838866312 (14h59m54.428211299s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838893975 (14h59m54.428238962s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.839056986 (14h59m54.428401974s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:55.935384249 (14h59m55.524729246s) [upload=15536]: Next returned nil segment Aug 8 00:39:55.935402815 (14h59m55.524747802s) [upload=15536]: Waiting for error group managing segments... Aug 8 00:39:56.226788010 (14h59m55.816132997s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.265710491 (14h59m55.855055478s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.326363980 (14h59m55.915708967s) [noise=true , address=64.49.13.169:4709 , piece_id=PIPH24KH4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773039652 (14h59m56.362384679s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773213735 (14h59m56.362558732s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.782851711 (14h59m56.372196698s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854376586 (14h59m56.443721573s) [noise=true , address=104.36.82.46:4484 , piece_id=BMHSO2DIXVRRHUZRHWUQ24TXMMTRMVX34SVDUUUMRMKPSUQG3APQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854703231 (14h59m56.444048218s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.914818370 (14h59m56.504163357s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.915843802 (14h59m56.505188849s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.942005621 (14h59m56.531350608s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.981103286 (14h59m56.570448273s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.095564053 (14h59m56.68490904s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216182613 (14h59m56.8055276s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216206869 (14h59m56.805551856s) [seg_pos={0 0}, upload=15536]: Segment reached optimal threshold of 14 pieces. Aug 8 00:39:57.216406632 (14h59m56.805751619s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bN25eEEjxCWKJSr8qj7moDJxArgfsjzPTdBcLdfvoeyxiqPpc): protocol: expected piece hash; context canceled; EOF Aug 8 00:39:57.216460465 (14h59m56.805805452s) [noise=true , address=104.36.82.5:4869 , piece_id=JBCDCG676BBV6U2GAVWDQIWYXKKQGW2DKK6MDY74LNPGP4K3CTQA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection Aug 8 00:39:57.216644467 (14h59m56.805989454s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD): protocol: read tcp 188.166.225.170:50728->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216665567 (14h59m56.806010554s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: read tcp 188.166.225.170:45052->104.36.82.61:4383: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216679534 (14h59m56.806024521s) [seg_pos={0 0}, upload=15536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:39:57.216739699 (14h59m56.806084686s) [upload=15536]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:39:57.216743286 (14h59m56.806088273s) [upload=15536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:39:57.516798834 (14h59m57.106143821s) [upload=15536]: Done uploading segments... Aug 8 00:39:57.516817129 (14h59m57.106162116s) [upload=15536]: Upload finished. err: <nil> Aug 8 00:40:09.579830255 (15h0m9.169175242s) [upload=15541]: Starting upload Aug 8 00:40:09.579877025 (15h0m9.169222012s) [upload=15541]: Done starting upload Aug 8 00:40:09.579941739 (15h0m9.169286726s) [upload=15541]: Uploading segments... Aug 8 00:40:09.580164405 (15h0m9.169509432s) [upload=15541]: Got next segment. Inline: false Aug 8 00:40:09.580219460 (15h0m9.169564477s) [upload=15541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:40:09.580251472 (15h0m9.169596489s) [upload=15541]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:40:09.797368422 (15h0m9.386713409s) [upload=15541]: Next segment err: metaclient: Too Many Requests Aug 8 00:40:09.797398149 (15h0m9.386743136s) [upload=15541]: Done uploading segments... Aug 8 00:40:09.797473614 (15h0m9.386818601s) [upload=15541]: Upload finished. err: metaclient: Too Many Requests Duration: 6.219342957s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.961111213s FAIL (2025-08-08 00:52:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9109253s) [upload=14753]: Starting upload Aug 7 23:54:04.489828037 (14h14m4.079173014s) [upload=14753]: Done starting upload Aug 7 23:54:04.489877612 (14h14m4.079222589s) [upload=14753]: Uploading segments... Aug 7 23:54:04.489903933 (14h14m4.07924892s) [upload=14753]: Got next segment. Inline: true Aug 7 23:54:04.489907469 (14h14m4.079252446s) [upload=14753]: Waiting for error group managing segments... Aug 7 23:54:04.489939651 (14h14m4.079284638s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:04.489943298 (14h14m4.079288285s) [upload=14753]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:04.489945472 (14h14m4.079290459s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:04.778726062 (14h14m4.368071049s) [upload=14753]: Done uploading segments... Aug 7 23:54:04.778744989 (14h14m4.368089976s) [upload=14753]: Upload finished. err: <nil> Aug 7 23:55:14.199719855 (14h15m13.789064892s) [upload=14770]: Starting upload Aug 7 23:55:14.199777645 (14h15m13.789122632s) [upload=14770]: Done starting upload Aug 7 23:55:14.199824415 (14h15m13.789169392s) [upload=14770]: Uploading segments... Aug 7 23:55:14.199852819 (14h15m13.789197806s) [upload=14770]: Got next segment. Inline: true Aug 7 23:55:14.199856436 (14h15m13.789201423s) [upload=14770]: Waiting for error group managing segments... Aug 7 23:55:14.199889419 (14h15m13.789234406s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:14.199893557 (14h15m13.789238544s) [upload=14770]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:14.199895951 (14h15m13.789240938s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:13.542516296 (14h16m13.131861283s) [upload=14770]: Done uploading segments... Aug 7 23:56:13.542536334 (14h16m13.131881321s) [upload=14770]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:14.428677736 (14h16m14.018022723s) [upload=14788]: Starting upload Aug 7 23:56:14.428730978 (14h16m14.018075965s) [upload=14788]: Done starting upload Aug 7 23:56:14.428789289 (14h16m14.018134276s) [upload=14788]: Uploading segments... Aug 7 23:56:14.428819026 (14h16m14.018164013s) [upload=14788]: Got next segment. Inline: true Aug 7 23:56:14.428822873 (14h16m14.01816785s) [upload=14788]: Waiting for error group managing segments... Aug 7 23:56:14.428855215 (14h16m14.018200202s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:14.428858562 (14h16m14.018203549s) [upload=14788]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:14.428860736 (14h16m14.018205723s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:13.781984834 (14h17m13.371329821s) [upload=14788]: Done uploading segments... Aug 7 23:57:13.782003370 (14h17m13.371348357s) [upload=14788]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:29.079359478 (14h17m28.668704466s) [upload=14808]: Starting upload Aug 7 23:57:29.079416147 (14h17m28.668761134s) [upload=14808]: Done starting upload Aug 7 23:57:29.079461845 (14h17m28.668806822s) [upload=14808]: Uploading segments... Aug 7 23:57:29.079488876 (14h17m28.668833863s) [upload=14808]: Got next segment. Inline: true Aug 7 23:57:29.079492263 (14h17m28.66883725s) [upload=14808]: Waiting for error group managing segments... Aug 7 23:57:29.079523392 (14h17m28.668868379s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:29.079527049 (14h17m28.668872036s) [upload=14808]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:29.079529484 (14h17m28.668874461s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:28.439377526 (14h18m28.028722513s) [upload=14808]: Done uploading segments... Aug 7 23:58:28.439395721 (14h18m28.028740708s) [upload=14808]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:29.333669502 (14h18m28.923014489s) [upload=14825]: Starting upload Aug 7 23:58:29.333742581 (14h18m28.923087568s) [upload=14825]: Done starting upload Aug 7 23:58:29.333835740 (14h18m28.923180727s) [upload=14825]: Uploading segments... Aug 7 23:58:29.333878000 (14h18m28.923222987s) [upload=14825]: Got next segment. Inline: true Aug 7 23:58:29.333881998 (14h18m28.923226975s) [upload=14825]: Waiting for error group managing segments... Aug 7 23:58:29.333911455 (14h18m28.923256442s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:29.333917496 (14h18m28.923262473s) [upload=14825]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:29.333919741 (14h18m28.923264728s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:28.692415040 (14h19m28.281760027s) [upload=14825]: Done uploading segments... Aug 7 23:59:28.692444777 (14h19m28.281789764s) [upload=14825]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:29.562470053 (14h19m29.15181504s) [upload=14840]: Starting upload Aug 7 23:59:29.562538985 (14h19m29.151883972s) [upload=14840]: Done starting upload Aug 7 23:59:29.562614629 (14h19m29.151959616s) [upload=14840]: Uploading segments... Aug 7 23:59:29.562650589 (14h19m29.151995566s) [upload=14840]: Got next segment. Inline: true Aug 7 23:59:29.562655498 (14h19m29.152000485s) [upload=14840]: Waiting for error group managing segments... Aug 7 23:59:29.562686507 (14h19m29.152031494s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:29.562690264 (14h19m29.152035241s) [upload=14840]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:29.562692428 (14h19m29.152037405s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:29.822497550 (14h19m29.411842537s) [upload=14840]: Done uploading segments... Aug 7 23:59:29.822514693 (14h19m29.41185968s) [upload=14840]: Upload finished. err: <nil> Aug 8 00:00:35.820554363 (14h20m35.40989935s) [upload=14856]: Starting upload Aug 8 00:00:35.820610611 (14h20m35.409955598s) [upload=14856]: Done starting upload Aug 8 00:00:35.820669503 (14h20m35.4100145s) [upload=14856]: Uploading segments... Aug 8 00:00:35.820742333 (14h20m35.41008732s) [upload=14856]: Got next segment. Inline: true Aug 8 00:00:35.820747282 (14h20m35.410092269s) [upload=14856]: Waiting for error group managing segments... Aug 8 00:00:35.820788371 (14h20m35.410133358s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.820793050 (14h20m35.410138037s) [upload=14856]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.820795485 (14h20m35.410140472s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:36.084321816 (14h20m35.673666803s) [upload=14856]: Done uploading segments... Aug 8 00:00:36.084338618 (14h20m35.673683615s) [upload=14856]: Upload finished. err: <nil> Aug 8 00:01:14.496466774 (14h21m14.085811781s) [upload=14868]: Starting upload Aug 8 00:01:14.496556174 (14h21m14.085901162s) [upload=14868]: Done starting upload Aug 8 00:01:14.496609206 (14h21m14.085954183s) [upload=14868]: Uploading segments... Aug 8 00:01:14.496639985 (14h21m14.085984972s) [upload=14868]: Got next segment. Inline: true Aug 8 00:01:14.496643592 (14h21m14.085988579s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:01:14.496678638 (14h21m14.086023625s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:14.496682386 (14h21m14.086027373s) [upload=14868]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:14.496684681 (14h21m14.086029658s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:14.760681029 (14h21m14.350026016s) [upload=14868]: Done uploading segments... Aug 8 00:01:14.760702580 (14h21m14.350047567s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:01:50.577049881 (14h21m50.166394858s) [upload=14878]: Starting upload Aug 8 00:01:50.577100538 (14h21m50.166445525s) [upload=14878]: Done starting upload Aug 8 00:01:50.577180091 (14h21m50.166525078s) [upload=14878]: Uploading segments... Aug 8 00:01:50.577228012 (14h21m50.166573s) [upload=14878]: Got next segment. Inline: true Aug 8 00:01:50.577231469 (14h21m50.166576456s) [upload=14878]: Waiting for error group managing segments... Aug 8 00:01:50.577270633 (14h21m50.166615621s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:50.577274721 (14h21m50.166619708s) [upload=14878]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:50.577276906 (14h21m50.166621893s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:50.863237339 (14h21m50.452582326s) [upload=14878]: Done uploading segments... Aug 8 00:01:50.863254973 (14h21m50.452599961s) [upload=14878]: Upload finished. err: <nil> Aug 8 00:02:41.474648957 (14h22m41.063993944s) [upload=14899]: Starting upload Aug 8 00:02:41.474716297 (14h22m41.064061274s) [upload=14899]: Done starting upload Aug 8 00:02:41.474771743 (14h22m41.06411673s) [upload=14899]: Uploading segments... Aug 8 00:02:41.474801099 (14h22m41.064146086s) [upload=14899]: Got next segment. Inline: true Aug 8 00:02:41.474804966 (14h22m41.064149953s) [upload=14899]: Waiting for error group managing segments... Aug 8 00:02:41.474841336 (14h22m41.064186323s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:41.474845214 (14h22m41.064190201s) [upload=14899]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:41.474847558 (14h22m41.064192535s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:03:40.835731182 (14h23m40.425076189s) [upload=14899]: Done uploading segments... Aug 8 00:03:40.835778212 (14h23m40.425123199s) [upload=14899]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.961140099s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..........X.......X...X............X....X..................] duration: 8.017646952s FAIL (2025-08-08 00:52:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837172821 (14h59m54.426517798s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837385949 (14h59m54.426730936s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837507802 (14h59m54.426852779s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837640145 (14h59m54.426985122s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837819459 (14h59m54.427164446s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837962502 (14h59m54.427307479s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838112790 (14h59m54.427457767s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838238691 (14h59m54.427583678s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838393447 (14h59m54.427738434s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838490402 (14h59m54.427835389s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838660288 (14h59m54.428005275s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838681548 (14h59m54.428026525s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838866312 (14h59m54.428211299s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838893975 (14h59m54.428238962s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.839056986 (14h59m54.428401974s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:55.935384249 (14h59m55.524729246s) [upload=15536]: Next returned nil segment Aug 8 00:39:55.935402815 (14h59m55.524747802s) [upload=15536]: Waiting for error group managing segments... Aug 8 00:39:56.226788010 (14h59m55.816132997s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.265710491 (14h59m55.855055478s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.326363980 (14h59m55.915708967s) [noise=true , address=64.49.13.169:4709 , piece_id=PIPH24KH4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773039652 (14h59m56.362384679s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773213735 (14h59m56.362558732s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.782851711 (14h59m56.372196698s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854376586 (14h59m56.443721573s) [noise=true , address=104.36.82.46:4484 , piece_id=BMHSO2DIXVRRHUZRHWUQ24TXMMTRMVX34SVDUUUMRMKPSUQG3APQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854703231 (14h59m56.444048218s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.914818370 (14h59m56.504163357s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.915843802 (14h59m56.505188849s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.942005621 (14h59m56.531350608s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.981103286 (14h59m56.570448273s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.095564053 (14h59m56.68490904s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216182613 (14h59m56.8055276s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216206869 (14h59m56.805551856s) [seg_pos={0 0}, upload=15536]: Segment reached optimal threshold of 14 pieces. Aug 8 00:39:57.216406632 (14h59m56.805751619s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bN25eEEjxCWKJSr8qj7moDJxArgfsjzPTdBcLdfvoeyxiqPpc): protocol: expected piece hash; context canceled; EOF Aug 8 00:39:57.216460465 (14h59m56.805805452s) [noise=true , address=104.36.82.5:4869 , piece_id=JBCDCG676BBV6U2GAVWDQIWYXKKQGW2DKK6MDY74LNPGP4K3CTQA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection Aug 8 00:39:57.216644467 (14h59m56.805989454s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD): protocol: read tcp 188.166.225.170:50728->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216665567 (14h59m56.806010554s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: read tcp 188.166.225.170:45052->104.36.82.61:4383: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216679534 (14h59m56.806024521s) [seg_pos={0 0}, upload=15536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:39:57.216739699 (14h59m56.806084686s) [upload=15536]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:39:57.216743286 (14h59m56.806088273s) [upload=15536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:39:57.516798834 (14h59m57.106143821s) [upload=15536]: Done uploading segments... Aug 8 00:39:57.516817129 (14h59m57.106162116s) [upload=15536]: Upload finished. err: <nil> Aug 8 00:40:09.579830255 (15h0m9.169175242s) [upload=15541]: Starting upload Aug 8 00:40:09.579877025 (15h0m9.169222012s) [upload=15541]: Done starting upload Aug 8 00:40:09.579941739 (15h0m9.169286726s) [upload=15541]: Uploading segments... Aug 8 00:40:09.580164405 (15h0m9.169509432s) [upload=15541]: Got next segment. Inline: false Aug 8 00:40:09.580219460 (15h0m9.169564477s) [upload=15541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:40:09.580251472 (15h0m9.169596489s) [upload=15541]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:40:09.797368422 (15h0m9.386713409s) [upload=15541]: Next segment err: metaclient: Too Many Requests Aug 8 00:40:09.797398149 (15h0m9.386743136s) [upload=15541]: Done uploading segments... Aug 8 00:40:09.797473614 (15h0m9.386818601s) [upload=15541]: Upload finished. err: metaclient: Too Many Requests Duration: 8.017666069s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X..........X.......X...X............X....X.................] duration: 5.995894313s FAIL (2025-08-08 00:52:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837172821 (14h59m54.426517798s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837385949 (14h59m54.426730936s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837507802 (14h59m54.426852779s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837640145 (14h59m54.426985122s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837819459 (14h59m54.427164446s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837962502 (14h59m54.427307479s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838112790 (14h59m54.427457767s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838238691 (14h59m54.427583678s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838393447 (14h59m54.427738434s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838490402 (14h59m54.427835389s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838660288 (14h59m54.428005275s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838681548 (14h59m54.428026525s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838866312 (14h59m54.428211299s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838893975 (14h59m54.428238962s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.839056986 (14h59m54.428401974s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:55.935384249 (14h59m55.524729246s) [upload=15536]: Next returned nil segment Aug 8 00:39:55.935402815 (14h59m55.524747802s) [upload=15536]: Waiting for error group managing segments... Aug 8 00:39:56.226788010 (14h59m55.816132997s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.265710491 (14h59m55.855055478s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.326363980 (14h59m55.915708967s) [noise=true , address=64.49.13.169:4709 , piece_id=PIPH24KH4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773039652 (14h59m56.362384679s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773213735 (14h59m56.362558732s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.782851711 (14h59m56.372196698s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854376586 (14h59m56.443721573s) [noise=true , address=104.36.82.46:4484 , piece_id=BMHSO2DIXVRRHUZRHWUQ24TXMMTRMVX34SVDUUUMRMKPSUQG3APQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854703231 (14h59m56.444048218s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.914818370 (14h59m56.504163357s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.915843802 (14h59m56.505188849s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.942005621 (14h59m56.531350608s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.981103286 (14h59m56.570448273s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.095564053 (14h59m56.68490904s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216182613 (14h59m56.8055276s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216206869 (14h59m56.805551856s) [seg_pos={0 0}, upload=15536]: Segment reached optimal threshold of 14 pieces. Aug 8 00:39:57.216406632 (14h59m56.805751619s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bN25eEEjxCWKJSr8qj7moDJxArgfsjzPTdBcLdfvoeyxiqPpc): protocol: expected piece hash; context canceled; EOF Aug 8 00:39:57.216460465 (14h59m56.805805452s) [noise=true , address=104.36.82.5:4869 , piece_id=JBCDCG676BBV6U2GAVWDQIWYXKKQGW2DKK6MDY74LNPGP4K3CTQA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection Aug 8 00:39:57.216644467 (14h59m56.805989454s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD): protocol: read tcp 188.166.225.170:50728->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216665567 (14h59m56.806010554s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: read tcp 188.166.225.170:45052->104.36.82.61:4383: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216679534 (14h59m56.806024521s) [seg_pos={0 0}, upload=15536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:39:57.216739699 (14h59m56.806084686s) [upload=15536]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:39:57.216743286 (14h59m56.806088273s) [upload=15536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:39:57.516798834 (14h59m57.106143821s) [upload=15536]: Done uploading segments... Aug 8 00:39:57.516817129 (14h59m57.106162116s) [upload=15536]: Upload finished. err: <nil> Aug 8 00:40:09.579830255 (15h0m9.169175242s) [upload=15541]: Starting upload Aug 8 00:40:09.579877025 (15h0m9.169222012s) [upload=15541]: Done starting upload Aug 8 00:40:09.579941739 (15h0m9.169286726s) [upload=15541]: Uploading segments... Aug 8 00:40:09.580164405 (15h0m9.169509432s) [upload=15541]: Got next segment. Inline: false Aug 8 00:40:09.580219460 (15h0m9.169564477s) [upload=15541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:40:09.580251472 (15h0m9.169596489s) [upload=15541]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:40:09.797368422 (15h0m9.386713409s) [upload=15541]: Next segment err: metaclient: Too Many Requests Aug 8 00:40:09.797398149 (15h0m9.386743136s) [upload=15541]: Done uploading segments... Aug 8 00:40:09.797473614 (15h0m9.386818601s) [upload=15541]: Upload finished. err: metaclient: Too Many Requests Duration: 5.995953627s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.828843858s FAIL (2025-08-08 00:51:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9109253s) [upload=14753]: Starting upload Aug 7 23:54:04.489828037 (14h14m4.079173014s) [upload=14753]: Done starting upload Aug 7 23:54:04.489877612 (14h14m4.079222589s) [upload=14753]: Uploading segments... Aug 7 23:54:04.489903933 (14h14m4.07924892s) [upload=14753]: Got next segment. Inline: true Aug 7 23:54:04.489907469 (14h14m4.079252446s) [upload=14753]: Waiting for error group managing segments... Aug 7 23:54:04.489939651 (14h14m4.079284638s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:04.489943298 (14h14m4.079288285s) [upload=14753]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:04.489945472 (14h14m4.079290459s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:04.778726062 (14h14m4.368071049s) [upload=14753]: Done uploading segments... Aug 7 23:54:04.778744989 (14h14m4.368089976s) [upload=14753]: Upload finished. err: <nil> Aug 7 23:55:14.199719855 (14h15m13.789064892s) [upload=14770]: Starting upload Aug 7 23:55:14.199777645 (14h15m13.789122632s) [upload=14770]: Done starting upload Aug 7 23:55:14.199824415 (14h15m13.789169392s) [upload=14770]: Uploading segments... Aug 7 23:55:14.199852819 (14h15m13.789197806s) [upload=14770]: Got next segment. Inline: true Aug 7 23:55:14.199856436 (14h15m13.789201423s) [upload=14770]: Waiting for error group managing segments... Aug 7 23:55:14.199889419 (14h15m13.789234406s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:14.199893557 (14h15m13.789238544s) [upload=14770]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:14.199895951 (14h15m13.789240938s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:13.542516296 (14h16m13.131861283s) [upload=14770]: Done uploading segments... Aug 7 23:56:13.542536334 (14h16m13.131881321s) [upload=14770]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:14.428677736 (14h16m14.018022723s) [upload=14788]: Starting upload Aug 7 23:56:14.428730978 (14h16m14.018075965s) [upload=14788]: Done starting upload Aug 7 23:56:14.428789289 (14h16m14.018134276s) [upload=14788]: Uploading segments... Aug 7 23:56:14.428819026 (14h16m14.018164013s) [upload=14788]: Got next segment. Inline: true Aug 7 23:56:14.428822873 (14h16m14.01816785s) [upload=14788]: Waiting for error group managing segments... Aug 7 23:56:14.428855215 (14h16m14.018200202s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:14.428858562 (14h16m14.018203549s) [upload=14788]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:14.428860736 (14h16m14.018205723s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:13.781984834 (14h17m13.371329821s) [upload=14788]: Done uploading segments... Aug 7 23:57:13.782003370 (14h17m13.371348357s) [upload=14788]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:29.079359478 (14h17m28.668704466s) [upload=14808]: Starting upload Aug 7 23:57:29.079416147 (14h17m28.668761134s) [upload=14808]: Done starting upload Aug 7 23:57:29.079461845 (14h17m28.668806822s) [upload=14808]: Uploading segments... Aug 7 23:57:29.079488876 (14h17m28.668833863s) [upload=14808]: Got next segment. Inline: true Aug 7 23:57:29.079492263 (14h17m28.66883725s) [upload=14808]: Waiting for error group managing segments... Aug 7 23:57:29.079523392 (14h17m28.668868379s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:29.079527049 (14h17m28.668872036s) [upload=14808]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:29.079529484 (14h17m28.668874461s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:28.439377526 (14h18m28.028722513s) [upload=14808]: Done uploading segments... Aug 7 23:58:28.439395721 (14h18m28.028740708s) [upload=14808]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:29.333669502 (14h18m28.923014489s) [upload=14825]: Starting upload Aug 7 23:58:29.333742581 (14h18m28.923087568s) [upload=14825]: Done starting upload Aug 7 23:58:29.333835740 (14h18m28.923180727s) [upload=14825]: Uploading segments... Aug 7 23:58:29.333878000 (14h18m28.923222987s) [upload=14825]: Got next segment. Inline: true Aug 7 23:58:29.333881998 (14h18m28.923226975s) [upload=14825]: Waiting for error group managing segments... Aug 7 23:58:29.333911455 (14h18m28.923256442s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:29.333917496 (14h18m28.923262473s) [upload=14825]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:29.333919741 (14h18m28.923264728s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:28.692415040 (14h19m28.281760027s) [upload=14825]: Done uploading segments... Aug 7 23:59:28.692444777 (14h19m28.281789764s) [upload=14825]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:29.562470053 (14h19m29.15181504s) [upload=14840]: Starting upload Aug 7 23:59:29.562538985 (14h19m29.151883972s) [upload=14840]: Done starting upload Aug 7 23:59:29.562614629 (14h19m29.151959616s) [upload=14840]: Uploading segments... Aug 7 23:59:29.562650589 (14h19m29.151995566s) [upload=14840]: Got next segment. Inline: true Aug 7 23:59:29.562655498 (14h19m29.152000485s) [upload=14840]: Waiting for error group managing segments... Aug 7 23:59:29.562686507 (14h19m29.152031494s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:29.562690264 (14h19m29.152035241s) [upload=14840]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:29.562692428 (14h19m29.152037405s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:29.822497550 (14h19m29.411842537s) [upload=14840]: Done uploading segments... Aug 7 23:59:29.822514693 (14h19m29.41185968s) [upload=14840]: Upload finished. err: <nil> Aug 8 00:00:35.820554363 (14h20m35.40989935s) [upload=14856]: Starting upload Aug 8 00:00:35.820610611 (14h20m35.409955598s) [upload=14856]: Done starting upload Aug 8 00:00:35.820669503 (14h20m35.4100145s) [upload=14856]: Uploading segments... Aug 8 00:00:35.820742333 (14h20m35.41008732s) [upload=14856]: Got next segment. Inline: true Aug 8 00:00:35.820747282 (14h20m35.410092269s) [upload=14856]: Waiting for error group managing segments... Aug 8 00:00:35.820788371 (14h20m35.410133358s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.820793050 (14h20m35.410138037s) [upload=14856]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.820795485 (14h20m35.410140472s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:36.084321816 (14h20m35.673666803s) [upload=14856]: Done uploading segments... Aug 8 00:00:36.084338618 (14h20m35.673683615s) [upload=14856]: Upload finished. err: <nil> Aug 8 00:01:14.496466774 (14h21m14.085811781s) [upload=14868]: Starting upload Aug 8 00:01:14.496556174 (14h21m14.085901162s) [upload=14868]: Done starting upload Aug 8 00:01:14.496609206 (14h21m14.085954183s) [upload=14868]: Uploading segments... Aug 8 00:01:14.496639985 (14h21m14.085984972s) [upload=14868]: Got next segment. Inline: true Aug 8 00:01:14.496643592 (14h21m14.085988579s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:01:14.496678638 (14h21m14.086023625s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:14.496682386 (14h21m14.086027373s) [upload=14868]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:14.496684681 (14h21m14.086029658s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:14.760681029 (14h21m14.350026016s) [upload=14868]: Done uploading segments... Aug 8 00:01:14.760702580 (14h21m14.350047567s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:01:50.577049881 (14h21m50.166394858s) [upload=14878]: Starting upload Aug 8 00:01:50.577100538 (14h21m50.166445525s) [upload=14878]: Done starting upload Aug 8 00:01:50.577180091 (14h21m50.166525078s) [upload=14878]: Uploading segments... Aug 8 00:01:50.577228012 (14h21m50.166573s) [upload=14878]: Got next segment. Inline: true Aug 8 00:01:50.577231469 (14h21m50.166576456s) [upload=14878]: Waiting for error group managing segments... Aug 8 00:01:50.577270633 (14h21m50.166615621s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:50.577274721 (14h21m50.166619708s) [upload=14878]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:50.577276906 (14h21m50.166621893s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:50.863237339 (14h21m50.452582326s) [upload=14878]: Done uploading segments... Aug 8 00:01:50.863254973 (14h21m50.452599961s) [upload=14878]: Upload finished. err: <nil> Aug 8 00:02:41.474648957 (14h22m41.063993944s) [upload=14899]: Starting upload Aug 8 00:02:41.474716297 (14h22m41.064061274s) [upload=14899]: Done starting upload Aug 8 00:02:41.474771743 (14h22m41.06411673s) [upload=14899]: Uploading segments... Aug 8 00:02:41.474801099 (14h22m41.064146086s) [upload=14899]: Got next segment. Inline: true Aug 8 00:02:41.474804966 (14h22m41.064149953s) [upload=14899]: Waiting for error group managing segments... Aug 8 00:02:41.474841336 (14h22m41.064186323s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:41.474845214 (14h22m41.064190201s) [upload=14899]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:41.474847558 (14h22m41.064192535s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:03:40.835731182 (14h23m40.425076189s) [upload=14899]: Done uploading segments... Aug 8 00:03:40.835778212 (14h23m40.425123199s) [upload=14899]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.828868124s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................X.........................................X.] duration: 2.04715414s FAIL (2025-08-08 00:51:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== stItem_ObjectCommit Aug 8 00:42:17.713078039 (15h2m17.302423046s) [upload=15574]: Done uploading segments... Aug 8 00:42:17.713100702 (15h2m17.30244574s) [upload=15574]: Upload finished. err: <nil> Aug 8 00:42:36.033866449 (15h2m35.623211436s) [upload=15582]: Starting upload Aug 8 00:42:36.033918057 (15h2m35.623263044s) [upload=15582]: Done starting upload Aug 8 00:42:36.033968163 (15h2m35.623313141s) [upload=15582]: Uploading segments... Aug 8 00:42:36.033998922 (15h2m35.623343909s) [upload=15582]: Got next segment. Inline: true Aug 8 00:42:36.034002399 (15h2m35.623347386s) [upload=15582]: Waiting for error group managing segments... Aug 8 00:42:36.034039881 (15h2m35.623384868s) [upload=15582]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:42:36.034043698 (15h2m35.623388685s) [upload=15582]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:42:36.034046063 (15h2m35.62339105s) [upload=15582]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:42:36.326408369 (15h2m35.915753356s) [upload=15582]: Done uploading segments... Aug 8 00:42:36.326427806 (15h2m35.915772823s) [upload=15582]: Upload finished. err: <nil> Aug 8 00:43:50.785976974 (15h3m50.375321961s) [upload=15600]: Starting upload Aug 8 00:43:50.786021608 (15h3m50.375366585s) [upload=15600]: Done starting upload Aug 8 00:43:50.786068328 (15h3m50.375413315s) [upload=15600]: Uploading segments... Aug 8 00:43:50.786096292 (15h3m50.375441269s) [upload=15600]: Got next segment. Inline: true Aug 8 00:43:50.786099928 (15h3m50.375444915s) [upload=15600]: Waiting for error group managing segments... Aug 8 00:43:50.786133022 (15h3m50.375478009s) [upload=15600]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:43:50.786136840 (15h3m50.375481817s) [upload=15600]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:43:50.786139374 (15h3m50.375484351s) [upload=15600]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:43:51.071276196 (15h3m50.660621183s) [upload=15600]: Done uploading segments... Aug 8 00:43:51.071294140 (15h3m50.660639127s) [upload=15600]: Upload finished. err: <nil> Aug 8 00:44:38.932060864 (15h4m38.521405851s) [upload=15614]: Starting upload Aug 8 00:44:38.932105570 (15h4m38.521450547s) [upload=15614]: Done starting upload Aug 8 00:44:38.932150475 (15h4m38.521495462s) [upload=15614]: Uploading segments... Aug 8 00:44:38.932180272 (15h4m38.521525249s) [upload=15614]: Got next segment. Inline: true Aug 8 00:44:38.932183829 (15h4m38.521528816s) [upload=15614]: Waiting for error group managing segments... Aug 8 00:44:38.932214318 (15h4m38.521559305s) [upload=15614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:44:38.932218205 (15h4m38.521563192s) [upload=15614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:44:38.932220419 (15h4m38.521565396s) [upload=15614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:44:39.257010779 (15h4m38.846355757s) [upload=15614]: Done uploading segments... Aug 8 00:44:39.257027903 (15h4m38.84637289s) [upload=15614]: Upload finished. err: <nil> Aug 8 00:44:55.671929119 (15h4m55.261274106s) [upload=15619]: Starting upload Aug 8 00:44:55.671974105 (15h4m55.261319092s) [upload=15619]: Done starting upload Aug 8 00:44:55.672033319 (15h4m55.261378296s) [upload=15619]: Uploading segments... Aug 8 00:44:55.672073296 (15h4m55.261418273s) [upload=15619]: Got next segment. Inline: true Aug 8 00:44:55.672077514 (15h4m55.261422501s) [upload=15619]: Waiting for error group managing segments... Aug 8 00:44:55.672111548 (15h4m55.261456535s) [upload=15619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:44:55.672115085 (15h4m55.261460072s) [upload=15619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:44:55.672117520 (15h4m55.261462507s) [upload=15619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:44:55.946154002 (15h4m55.535498989s) [upload=15619]: Done uploading segments... Aug 8 00:44:55.946180583 (15h4m55.53552557s) [upload=15619]: Upload finished. err: <nil> Aug 8 00:45:51.335687032 (15h5m50.925032019s) [upload=15634]: Starting upload Aug 8 00:45:51.335741947 (15h5m50.925086934s) [upload=15634]: Done starting upload Aug 8 00:45:51.335838182 (15h5m50.925183169s) [upload=15634]: Uploading segments... Aug 8 00:45:51.335870213 (15h5m50.9252152s) [upload=15634]: Got next segment. Inline: true Aug 8 00:45:51.335873930 (15h5m50.925218917s) [upload=15634]: Waiting for error group managing segments... Aug 8 00:45:51.335917033 (15h5m50.92526202s) [upload=15634]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:45:51.335921060 (15h5m50.925266047s) [upload=15634]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:45:51.335923345 (15h5m50.925268332s) [upload=15634]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:45:51.622624488 (15h5m51.211969475s) [upload=15634]: Done uploading segments... Aug 8 00:45:51.622650428 (15h5m51.211995415s) [upload=15634]: Upload finished. err: <nil> Aug 8 00:46:56.158030766 (15h6m55.747375753s) [upload=15655]: Starting upload Aug 8 00:46:56.158075261 (15h6m55.747420248s) [upload=15655]: Done starting upload Aug 8 00:46:56.158127792 (15h6m55.747472779s) [upload=15655]: Uploading segments... Aug 8 00:46:56.158159252 (15h6m55.747504229s) [upload=15655]: Got next segment. Inline: true Aug 8 00:46:56.158162568 (15h6m55.747507555s) [upload=15655]: Waiting for error group managing segments... Aug 8 00:46:56.158193087 (15h6m55.747538074s) [upload=15655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:46:56.158196653 (15h6m55.74754164s) [upload=15655]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:46:56.158199098 (15h6m55.747544075s) [upload=15655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:46:56.421913161 (15h6m56.011258168s) [upload=15655]: Done uploading segments... Aug 8 00:46:56.421936716 (15h6m56.011281703s) [upload=15655]: Upload finished. err: <nil> Aug 8 00:48:03.458104986 (15h8m3.047449973s) [upload=15670]: Starting upload Aug 8 00:48:03.458166774 (15h8m3.047511761s) [upload=15670]: Done starting upload Aug 8 00:48:03.458226910 (15h8m3.047571887s) [upload=15670]: Uploading segments... Aug 8 00:48:03.458331268 (15h8m3.047676255s) [upload=15670]: Got next segment. Inline: true Aug 8 00:48:03.458336880 (15h8m3.047681857s) [upload=15670]: Waiting for error group managing segments... Aug 8 00:48:03.458383799 (15h8m3.047728786s) [upload=15670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:48:03.458388839 (15h8m3.047733827s) [upload=15670]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:48:03.458391154 (15h8m3.047736141s) [upload=15670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:48:03.756480234 (15h8m3.345825221s) [upload=15670]: Done uploading segments... Aug 8 00:48:03.756500813 (15h8m3.3458458s) [upload=15670]: Upload finished. err: <nil> Aug 8 00:49:01.779484346 (15h9m1.368829333s) [upload=15683]: Starting upload Aug 8 00:49:01.779541455 (15h9m1.368886442s) [upload=15683]: Done starting upload Aug 8 00:49:01.779591250 (15h9m1.368936238s) [upload=15683]: Uploading segments... Aug 8 00:49:01.779632399 (15h9m1.368977386s) [upload=15683]: Got next segment. Inline: true Aug 8 00:49:01.779637188 (15h9m1.368982185s) [upload=15683]: Waiting for error group managing segments... Aug 8 00:49:01.779673187 (15h9m1.369018174s) [upload=15683]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:49:01.779677375 (15h9m1.369022362s) [upload=15683]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:49:01.779680040 (15h9m1.369025027s) [upload=15683]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:49:02.060683371 (15h9m1.650028378s) [upload=15683]: Done uploading segments... Aug 8 00:49:02.060710513 (15h9m1.6500555s) [upload=15683]: Upload finished. err: <nil> Aug 8 00:50:06.726842474 (15h10m6.316187461s) [upload=15705]: Starting upload Aug 8 00:50:06.726885055 (15h10m6.316230032s) [upload=15705]: Done starting upload Aug 8 00:50:06.726932025 (15h10m6.316277012s) [upload=15705]: Uploading segments... Aug 8 00:50:06.726962704 (15h10m6.316307691s) [upload=15705]: Got next segment. Inline: true Aug 8 00:50:06.726966831 (15h10m6.316311808s) [upload=15705]: Waiting for error group managing segments... Aug 8 00:50:06.726999844 (15h10m6.316344831s) [upload=15705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:50:06.727003823 (15h10m6.31634881s) [upload=15705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:50:06.727006187 (15h10m6.316351164s) [upload=15705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:50:07.002622120 (15h10m6.591967127s) [upload=15705]: Done uploading segments... Aug 8 00:50:07.002639403 (15h10m6.59198438s) [upload=15705]: Upload finished. err: <nil> Aug 8 00:51:18.800365702 (15h11m18.389710689s) [upload=15723]: Starting upload Aug 8 00:51:18.800416448 (15h11m18.389761435s) [upload=15723]: Done starting upload Aug 8 00:51:18.800490300 (15h11m18.389835277s) [upload=15723]: Uploading segments... Aug 8 00:51:18.800624426 (15h11m18.389969413s) [upload=15723]: Got next segment. Inline: true Aug 8 00:51:18.800629807 (15h11m18.389974794s) [upload=15723]: Waiting for error group managing segments... Aug 8 00:51:18.800677268 (15h11m18.390022255s) [upload=15723]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:51:18.800682698 (15h11m18.390027685s) [upload=15723]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:51:18.800685353 (15h11m18.39003034s) [upload=15723]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:51:19.015137684 (15h11m18.604482671s) [upload=15723]: Done uploading segments... Aug 8 00:51:19.015155639 (15h11m18.604500616s) [upload=15723]: Upload finished. err: metaclient: Too Many Requests Duration: 2.04718133s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X..........X.......X...X............X....X................] duration: 5.966818805s FAIL (2025-08-08 00:51:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837172821 (14h59m54.426517798s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837385949 (14h59m54.426730936s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837507802 (14h59m54.426852779s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837640145 (14h59m54.426985122s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837819459 (14h59m54.427164446s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837962502 (14h59m54.427307479s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838112790 (14h59m54.427457767s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838238691 (14h59m54.427583678s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838393447 (14h59m54.427738434s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838490402 (14h59m54.427835389s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838660288 (14h59m54.428005275s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838681548 (14h59m54.428026525s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838866312 (14h59m54.428211299s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838893975 (14h59m54.428238962s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.839056986 (14h59m54.428401974s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:55.935384249 (14h59m55.524729246s) [upload=15536]: Next returned nil segment Aug 8 00:39:55.935402815 (14h59m55.524747802s) [upload=15536]: Waiting for error group managing segments... Aug 8 00:39:56.226788010 (14h59m55.816132997s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.265710491 (14h59m55.855055478s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.326363980 (14h59m55.915708967s) [noise=true , address=64.49.13.169:4709 , piece_id=PIPH24KH4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773039652 (14h59m56.362384679s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773213735 (14h59m56.362558732s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.782851711 (14h59m56.372196698s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854376586 (14h59m56.443721573s) [noise=true , address=104.36.82.46:4484 , piece_id=BMHSO2DIXVRRHUZRHWUQ24TXMMTRMVX34SVDUUUMRMKPSUQG3APQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854703231 (14h59m56.444048218s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.914818370 (14h59m56.504163357s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.915843802 (14h59m56.505188849s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.942005621 (14h59m56.531350608s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.981103286 (14h59m56.570448273s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.095564053 (14h59m56.68490904s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216182613 (14h59m56.8055276s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216206869 (14h59m56.805551856s) [seg_pos={0 0}, upload=15536]: Segment reached optimal threshold of 14 pieces. Aug 8 00:39:57.216406632 (14h59m56.805751619s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bN25eEEjxCWKJSr8qj7moDJxArgfsjzPTdBcLdfvoeyxiqPpc): protocol: expected piece hash; context canceled; EOF Aug 8 00:39:57.216460465 (14h59m56.805805452s) [noise=true , address=104.36.82.5:4869 , piece_id=JBCDCG676BBV6U2GAVWDQIWYXKKQGW2DKK6MDY74LNPGP4K3CTQA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection Aug 8 00:39:57.216644467 (14h59m56.805989454s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD): protocol: read tcp 188.166.225.170:50728->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216665567 (14h59m56.806010554s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: read tcp 188.166.225.170:45052->104.36.82.61:4383: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216679534 (14h59m56.806024521s) [seg_pos={0 0}, upload=15536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:39:57.216739699 (14h59m56.806084686s) [upload=15536]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:39:57.216743286 (14h59m56.806088273s) [upload=15536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:39:57.516798834 (14h59m57.106143821s) [upload=15536]: Done uploading segments... Aug 8 00:39:57.516817129 (14h59m57.106162116s) [upload=15536]: Upload finished. err: <nil> Aug 8 00:40:09.579830255 (15h0m9.169175242s) [upload=15541]: Starting upload Aug 8 00:40:09.579877025 (15h0m9.169222012s) [upload=15541]: Done starting upload Aug 8 00:40:09.579941739 (15h0m9.169286726s) [upload=15541]: Uploading segments... Aug 8 00:40:09.580164405 (15h0m9.169509432s) [upload=15541]: Got next segment. Inline: false Aug 8 00:40:09.580219460 (15h0m9.169564477s) [upload=15541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:40:09.580251472 (15h0m9.169596489s) [upload=15541]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:40:09.797368422 (15h0m9.386713409s) [upload=15541]: Next segment err: metaclient: Too Many Requests Aug 8 00:40:09.797398149 (15h0m9.386743136s) [upload=15541]: Done uploading segments... Aug 8 00:40:09.797473614 (15h0m9.386818601s) [upload=15541]: Upload finished. err: metaclient: Too Many Requests Duration: 5.966866776s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................X.........................................X] duration: 867.398739ms FAIL (2025-08-08 00:51:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== stItem_ObjectCommit Aug 8 00:42:17.713078039 (15h2m17.302423046s) [upload=15574]: Done uploading segments... Aug 8 00:42:17.713100702 (15h2m17.30244574s) [upload=15574]: Upload finished. err: <nil> Aug 8 00:42:36.033866449 (15h2m35.623211436s) [upload=15582]: Starting upload Aug 8 00:42:36.033918057 (15h2m35.623263044s) [upload=15582]: Done starting upload Aug 8 00:42:36.033968163 (15h2m35.623313141s) [upload=15582]: Uploading segments... Aug 8 00:42:36.033998922 (15h2m35.623343909s) [upload=15582]: Got next segment. Inline: true Aug 8 00:42:36.034002399 (15h2m35.623347386s) [upload=15582]: Waiting for error group managing segments... Aug 8 00:42:36.034039881 (15h2m35.623384868s) [upload=15582]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:42:36.034043698 (15h2m35.623388685s) [upload=15582]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:42:36.034046063 (15h2m35.62339105s) [upload=15582]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:42:36.326408369 (15h2m35.915753356s) [upload=15582]: Done uploading segments... Aug 8 00:42:36.326427806 (15h2m35.915772823s) [upload=15582]: Upload finished. err: <nil> Aug 8 00:43:50.785976974 (15h3m50.375321961s) [upload=15600]: Starting upload Aug 8 00:43:50.786021608 (15h3m50.375366585s) [upload=15600]: Done starting upload Aug 8 00:43:50.786068328 (15h3m50.375413315s) [upload=15600]: Uploading segments... Aug 8 00:43:50.786096292 (15h3m50.375441269s) [upload=15600]: Got next segment. Inline: true Aug 8 00:43:50.786099928 (15h3m50.375444915s) [upload=15600]: Waiting for error group managing segments... Aug 8 00:43:50.786133022 (15h3m50.375478009s) [upload=15600]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:43:50.786136840 (15h3m50.375481817s) [upload=15600]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:43:50.786139374 (15h3m50.375484351s) [upload=15600]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:43:51.071276196 (15h3m50.660621183s) [upload=15600]: Done uploading segments... Aug 8 00:43:51.071294140 (15h3m50.660639127s) [upload=15600]: Upload finished. err: <nil> Aug 8 00:44:38.932060864 (15h4m38.521405851s) [upload=15614]: Starting upload Aug 8 00:44:38.932105570 (15h4m38.521450547s) [upload=15614]: Done starting upload Aug 8 00:44:38.932150475 (15h4m38.521495462s) [upload=15614]: Uploading segments... Aug 8 00:44:38.932180272 (15h4m38.521525249s) [upload=15614]: Got next segment. Inline: true Aug 8 00:44:38.932183829 (15h4m38.521528816s) [upload=15614]: Waiting for error group managing segments... Aug 8 00:44:38.932214318 (15h4m38.521559305s) [upload=15614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:44:38.932218205 (15h4m38.521563192s) [upload=15614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:44:38.932220419 (15h4m38.521565396s) [upload=15614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:44:39.257010779 (15h4m38.846355757s) [upload=15614]: Done uploading segments... Aug 8 00:44:39.257027903 (15h4m38.84637289s) [upload=15614]: Upload finished. err: <nil> Aug 8 00:44:55.671929119 (15h4m55.261274106s) [upload=15619]: Starting upload Aug 8 00:44:55.671974105 (15h4m55.261319092s) [upload=15619]: Done starting upload Aug 8 00:44:55.672033319 (15h4m55.261378296s) [upload=15619]: Uploading segments... Aug 8 00:44:55.672073296 (15h4m55.261418273s) [upload=15619]: Got next segment. Inline: true Aug 8 00:44:55.672077514 (15h4m55.261422501s) [upload=15619]: Waiting for error group managing segments... Aug 8 00:44:55.672111548 (15h4m55.261456535s) [upload=15619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:44:55.672115085 (15h4m55.261460072s) [upload=15619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:44:55.672117520 (15h4m55.261462507s) [upload=15619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:44:55.946154002 (15h4m55.535498989s) [upload=15619]: Done uploading segments... Aug 8 00:44:55.946180583 (15h4m55.53552557s) [upload=15619]: Upload finished. err: <nil> Aug 8 00:45:51.335687032 (15h5m50.925032019s) [upload=15634]: Starting upload Aug 8 00:45:51.335741947 (15h5m50.925086934s) [upload=15634]: Done starting upload Aug 8 00:45:51.335838182 (15h5m50.925183169s) [upload=15634]: Uploading segments... Aug 8 00:45:51.335870213 (15h5m50.9252152s) [upload=15634]: Got next segment. Inline: true Aug 8 00:45:51.335873930 (15h5m50.925218917s) [upload=15634]: Waiting for error group managing segments... Aug 8 00:45:51.335917033 (15h5m50.92526202s) [upload=15634]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:45:51.335921060 (15h5m50.925266047s) [upload=15634]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:45:51.335923345 (15h5m50.925268332s) [upload=15634]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:45:51.622624488 (15h5m51.211969475s) [upload=15634]: Done uploading segments... Aug 8 00:45:51.622650428 (15h5m51.211995415s) [upload=15634]: Upload finished. err: <nil> Aug 8 00:46:56.158030766 (15h6m55.747375753s) [upload=15655]: Starting upload Aug 8 00:46:56.158075261 (15h6m55.747420248s) [upload=15655]: Done starting upload Aug 8 00:46:56.158127792 (15h6m55.747472779s) [upload=15655]: Uploading segments... Aug 8 00:46:56.158159252 (15h6m55.747504229s) [upload=15655]: Got next segment. Inline: true Aug 8 00:46:56.158162568 (15h6m55.747507555s) [upload=15655]: Waiting for error group managing segments... Aug 8 00:46:56.158193087 (15h6m55.747538074s) [upload=15655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:46:56.158196653 (15h6m55.74754164s) [upload=15655]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:46:56.158199098 (15h6m55.747544075s) [upload=15655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:46:56.421913161 (15h6m56.011258168s) [upload=15655]: Done uploading segments... Aug 8 00:46:56.421936716 (15h6m56.011281703s) [upload=15655]: Upload finished. err: <nil> Aug 8 00:48:03.458104986 (15h8m3.047449973s) [upload=15670]: Starting upload Aug 8 00:48:03.458166774 (15h8m3.047511761s) [upload=15670]: Done starting upload Aug 8 00:48:03.458226910 (15h8m3.047571887s) [upload=15670]: Uploading segments... Aug 8 00:48:03.458331268 (15h8m3.047676255s) [upload=15670]: Got next segment. Inline: true Aug 8 00:48:03.458336880 (15h8m3.047681857s) [upload=15670]: Waiting for error group managing segments... Aug 8 00:48:03.458383799 (15h8m3.047728786s) [upload=15670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:48:03.458388839 (15h8m3.047733827s) [upload=15670]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:48:03.458391154 (15h8m3.047736141s) [upload=15670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:48:03.756480234 (15h8m3.345825221s) [upload=15670]: Done uploading segments... Aug 8 00:48:03.756500813 (15h8m3.3458458s) [upload=15670]: Upload finished. err: <nil> Aug 8 00:49:01.779484346 (15h9m1.368829333s) [upload=15683]: Starting upload Aug 8 00:49:01.779541455 (15h9m1.368886442s) [upload=15683]: Done starting upload Aug 8 00:49:01.779591250 (15h9m1.368936238s) [upload=15683]: Uploading segments... Aug 8 00:49:01.779632399 (15h9m1.368977386s) [upload=15683]: Got next segment. Inline: true Aug 8 00:49:01.779637188 (15h9m1.368982185s) [upload=15683]: Waiting for error group managing segments... Aug 8 00:49:01.779673187 (15h9m1.369018174s) [upload=15683]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:49:01.779677375 (15h9m1.369022362s) [upload=15683]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:49:01.779680040 (15h9m1.369025027s) [upload=15683]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:49:02.060683371 (15h9m1.650028378s) [upload=15683]: Done uploading segments... Aug 8 00:49:02.060710513 (15h9m1.6500555s) [upload=15683]: Upload finished. err: <nil> Aug 8 00:50:06.726842474 (15h10m6.316187461s) [upload=15705]: Starting upload Aug 8 00:50:06.726885055 (15h10m6.316230032s) [upload=15705]: Done starting upload Aug 8 00:50:06.726932025 (15h10m6.316277012s) [upload=15705]: Uploading segments... Aug 8 00:50:06.726962704 (15h10m6.316307691s) [upload=15705]: Got next segment. Inline: true Aug 8 00:50:06.726966831 (15h10m6.316311808s) [upload=15705]: Waiting for error group managing segments... Aug 8 00:50:06.726999844 (15h10m6.316344831s) [upload=15705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:50:06.727003823 (15h10m6.31634881s) [upload=15705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:50:06.727006187 (15h10m6.316351164s) [upload=15705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:50:07.002622120 (15h10m6.591967127s) [upload=15705]: Done uploading segments... Aug 8 00:50:07.002639403 (15h10m6.59198438s) [upload=15705]: Upload finished. err: <nil> Aug 8 00:51:18.800365702 (15h11m18.389710689s) [upload=15723]: Starting upload Aug 8 00:51:18.800416448 (15h11m18.389761435s) [upload=15723]: Done starting upload Aug 8 00:51:18.800490300 (15h11m18.389835277s) [upload=15723]: Uploading segments... Aug 8 00:51:18.800624426 (15h11m18.389969413s) [upload=15723]: Got next segment. Inline: true Aug 8 00:51:18.800629807 (15h11m18.389974794s) [upload=15723]: Waiting for error group managing segments... Aug 8 00:51:18.800677268 (15h11m18.390022255s) [upload=15723]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:51:18.800682698 (15h11m18.390027685s) [upload=15723]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:51:18.800685353 (15h11m18.39003034s) [upload=15723]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:51:19.015137684 (15h11m18.604482671s) [upload=15723]: Done uploading segments... Aug 8 00:51:19.015155639 (15h11m18.604500616s) [upload=15723]: Upload finished. err: metaclient: Too Many Requests Duration: 867.41986ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 1.874371443s FAIL (2025-08-08 00:51:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9109253s) [upload=14753]: Starting upload Aug 7 23:54:04.489828037 (14h14m4.079173014s) [upload=14753]: Done starting upload Aug 7 23:54:04.489877612 (14h14m4.079222589s) [upload=14753]: Uploading segments... Aug 7 23:54:04.489903933 (14h14m4.07924892s) [upload=14753]: Got next segment. Inline: true Aug 7 23:54:04.489907469 (14h14m4.079252446s) [upload=14753]: Waiting for error group managing segments... Aug 7 23:54:04.489939651 (14h14m4.079284638s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:04.489943298 (14h14m4.079288285s) [upload=14753]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:04.489945472 (14h14m4.079290459s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:04.778726062 (14h14m4.368071049s) [upload=14753]: Done uploading segments... Aug 7 23:54:04.778744989 (14h14m4.368089976s) [upload=14753]: Upload finished. err: <nil> Aug 7 23:55:14.199719855 (14h15m13.789064892s) [upload=14770]: Starting upload Aug 7 23:55:14.199777645 (14h15m13.789122632s) [upload=14770]: Done starting upload Aug 7 23:55:14.199824415 (14h15m13.789169392s) [upload=14770]: Uploading segments... Aug 7 23:55:14.199852819 (14h15m13.789197806s) [upload=14770]: Got next segment. Inline: true Aug 7 23:55:14.199856436 (14h15m13.789201423s) [upload=14770]: Waiting for error group managing segments... Aug 7 23:55:14.199889419 (14h15m13.789234406s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:14.199893557 (14h15m13.789238544s) [upload=14770]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:14.199895951 (14h15m13.789240938s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:13.542516296 (14h16m13.131861283s) [upload=14770]: Done uploading segments... Aug 7 23:56:13.542536334 (14h16m13.131881321s) [upload=14770]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:14.428677736 (14h16m14.018022723s) [upload=14788]: Starting upload Aug 7 23:56:14.428730978 (14h16m14.018075965s) [upload=14788]: Done starting upload Aug 7 23:56:14.428789289 (14h16m14.018134276s) [upload=14788]: Uploading segments... Aug 7 23:56:14.428819026 (14h16m14.018164013s) [upload=14788]: Got next segment. Inline: true Aug 7 23:56:14.428822873 (14h16m14.01816785s) [upload=14788]: Waiting for error group managing segments... Aug 7 23:56:14.428855215 (14h16m14.018200202s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:14.428858562 (14h16m14.018203549s) [upload=14788]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:14.428860736 (14h16m14.018205723s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:13.781984834 (14h17m13.371329821s) [upload=14788]: Done uploading segments... Aug 7 23:57:13.782003370 (14h17m13.371348357s) [upload=14788]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:29.079359478 (14h17m28.668704466s) [upload=14808]: Starting upload Aug 7 23:57:29.079416147 (14h17m28.668761134s) [upload=14808]: Done starting upload Aug 7 23:57:29.079461845 (14h17m28.668806822s) [upload=14808]: Uploading segments... Aug 7 23:57:29.079488876 (14h17m28.668833863s) [upload=14808]: Got next segment. Inline: true Aug 7 23:57:29.079492263 (14h17m28.66883725s) [upload=14808]: Waiting for error group managing segments... Aug 7 23:57:29.079523392 (14h17m28.668868379s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:29.079527049 (14h17m28.668872036s) [upload=14808]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:29.079529484 (14h17m28.668874461s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:28.439377526 (14h18m28.028722513s) [upload=14808]: Done uploading segments... Aug 7 23:58:28.439395721 (14h18m28.028740708s) [upload=14808]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:29.333669502 (14h18m28.923014489s) [upload=14825]: Starting upload Aug 7 23:58:29.333742581 (14h18m28.923087568s) [upload=14825]: Done starting upload Aug 7 23:58:29.333835740 (14h18m28.923180727s) [upload=14825]: Uploading segments... Aug 7 23:58:29.333878000 (14h18m28.923222987s) [upload=14825]: Got next segment. Inline: true Aug 7 23:58:29.333881998 (14h18m28.923226975s) [upload=14825]: Waiting for error group managing segments... Aug 7 23:58:29.333911455 (14h18m28.923256442s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:29.333917496 (14h18m28.923262473s) [upload=14825]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:29.333919741 (14h18m28.923264728s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:28.692415040 (14h19m28.281760027s) [upload=14825]: Done uploading segments... Aug 7 23:59:28.692444777 (14h19m28.281789764s) [upload=14825]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:29.562470053 (14h19m29.15181504s) [upload=14840]: Starting upload Aug 7 23:59:29.562538985 (14h19m29.151883972s) [upload=14840]: Done starting upload Aug 7 23:59:29.562614629 (14h19m29.151959616s) [upload=14840]: Uploading segments... Aug 7 23:59:29.562650589 (14h19m29.151995566s) [upload=14840]: Got next segment. Inline: true Aug 7 23:59:29.562655498 (14h19m29.152000485s) [upload=14840]: Waiting for error group managing segments... Aug 7 23:59:29.562686507 (14h19m29.152031494s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:29.562690264 (14h19m29.152035241s) [upload=14840]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:29.562692428 (14h19m29.152037405s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:29.822497550 (14h19m29.411842537s) [upload=14840]: Done uploading segments... Aug 7 23:59:29.822514693 (14h19m29.41185968s) [upload=14840]: Upload finished. err: <nil> Aug 8 00:00:35.820554363 (14h20m35.40989935s) [upload=14856]: Starting upload Aug 8 00:00:35.820610611 (14h20m35.409955598s) [upload=14856]: Done starting upload Aug 8 00:00:35.820669503 (14h20m35.4100145s) [upload=14856]: Uploading segments... Aug 8 00:00:35.820742333 (14h20m35.41008732s) [upload=14856]: Got next segment. Inline: true Aug 8 00:00:35.820747282 (14h20m35.410092269s) [upload=14856]: Waiting for error group managing segments... Aug 8 00:00:35.820788371 (14h20m35.410133358s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.820793050 (14h20m35.410138037s) [upload=14856]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.820795485 (14h20m35.410140472s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:36.084321816 (14h20m35.673666803s) [upload=14856]: Done uploading segments... Aug 8 00:00:36.084338618 (14h20m35.673683615s) [upload=14856]: Upload finished. err: <nil> Aug 8 00:01:14.496466774 (14h21m14.085811781s) [upload=14868]: Starting upload Aug 8 00:01:14.496556174 (14h21m14.085901162s) [upload=14868]: Done starting upload Aug 8 00:01:14.496609206 (14h21m14.085954183s) [upload=14868]: Uploading segments... Aug 8 00:01:14.496639985 (14h21m14.085984972s) [upload=14868]: Got next segment. Inline: true Aug 8 00:01:14.496643592 (14h21m14.085988579s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:01:14.496678638 (14h21m14.086023625s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:14.496682386 (14h21m14.086027373s) [upload=14868]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:14.496684681 (14h21m14.086029658s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:14.760681029 (14h21m14.350026016s) [upload=14868]: Done uploading segments... Aug 8 00:01:14.760702580 (14h21m14.350047567s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:01:50.577049881 (14h21m50.166394858s) [upload=14878]: Starting upload Aug 8 00:01:50.577100538 (14h21m50.166445525s) [upload=14878]: Done starting upload Aug 8 00:01:50.577180091 (14h21m50.166525078s) [upload=14878]: Uploading segments... Aug 8 00:01:50.577228012 (14h21m50.166573s) [upload=14878]: Got next segment. Inline: true Aug 8 00:01:50.577231469 (14h21m50.166576456s) [upload=14878]: Waiting for error group managing segments... Aug 8 00:01:50.577270633 (14h21m50.166615621s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:50.577274721 (14h21m50.166619708s) [upload=14878]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:50.577276906 (14h21m50.166621893s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:50.863237339 (14h21m50.452582326s) [upload=14878]: Done uploading segments... Aug 8 00:01:50.863254973 (14h21m50.452599961s) [upload=14878]: Upload finished. err: <nil> Aug 8 00:02:41.474648957 (14h22m41.063993944s) [upload=14899]: Starting upload Aug 8 00:02:41.474716297 (14h22m41.064061274s) [upload=14899]: Done starting upload Aug 8 00:02:41.474771743 (14h22m41.06411673s) [upload=14899]: Uploading segments... Aug 8 00:02:41.474801099 (14h22m41.064146086s) [upload=14899]: Got next segment. Inline: true Aug 8 00:02:41.474804966 (14h22m41.064149953s) [upload=14899]: Waiting for error group managing segments... Aug 8 00:02:41.474841336 (14h22m41.064186323s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:41.474845214 (14h22m41.064190201s) [upload=14899]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:41.474847558 (14h22m41.064192535s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:03:40.835731182 (14h23m40.425076189s) [upload=14899]: Done uploading segments... Aug 8 00:03:40.835778212 (14h23m40.425123199s) [upload=14899]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.874402081s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.24337074s FAIL (2025-08-08 00:50:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.246137011 (14h21m15.835481998s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.246560380 (14h21m15.835905367s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247408974 (14h21m15.836753961s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247678410 (14h21m15.837023397s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247919882 (14h21m15.837264869s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247943888 (14h21m15.837288875s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248293266 (14h21m15.837638253s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248420540 (14h21m15.837765527s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248557311 (14h21m15.837902288s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248758506 (14h21m15.838103493s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248897792 (14h21m15.838242779s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248989629 (14h21m15.838334616s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249186575 (14h21m15.838531552s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249273772 (14h21m15.838618759s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249459598 (14h21m15.838804585s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249717240 (14h21m15.839062227s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:17.413257645 (14h21m17.002602632s) [upload=14870]: Next returned nil segment Aug 8 00:01:17.413273324 (14h21m17.002618311s) [upload=14870]: Waiting for error group managing segments... Aug 8 00:01:17.710897294 (14h21m17.300242281s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:17.715058896 (14h21m17.304403883s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.126218161 (14h21m17.715563148s) [noise=true , address=79.127.201.215:4356 , piece_id=VWNEIHNI4KPPF4QLW36INIWTUI2CD6MZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.139272367 (14h21m17.728617354s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.221269031 (14h21m17.810614018s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.257350144 (14h21m17.846695121s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.258435089 (14h21m17.847780076s) [noise=true , address=104.36.82.44:4876 , piece_id=BA2G42OXQD3NLONUVD475S2BNC7T37T5PHPE3K3E5HJAJP4W5MKQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.287010923 (14h21m17.87635591s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.348793001 (14h21m17.938137988s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.378129459 (14h21m17.967474436s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.386414901 (14h21m17.975759888s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.516150823 (14h21m18.10549581s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.626647204 (14h21m18.215992191s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:19.042108811 (14h21m18.631453798s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:19.042136654 (14h21m18.631481641s) [seg_pos={0 0}, upload=14870]: Segment reached optimal threshold of 14 pieces. Aug 8 00:01:19.042375541 (14h21m18.631720518s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:19.042485912 (14h21m18.6318309s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: rpc: dial tcp 104.36.82.55:4209: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:01:19.042532361 (14h21m18.631877348s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqNLPRWY7fJRS6F9h4B4qVFv6U5jyWkCy3Um7jfjEdCShEjsD): protocol: write tcp 188.166.225.170:37056->104.36.82.20:4466: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:01:19.042616462 (14h21m18.631961449s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa): protocol: rpc: dial tcp 104.36.82.5:4844: operation was canceled; conn closed: op; piecestore: piecestore close: context canceled Aug 8 00:01:19.042628876 (14h21m18.631973863s) [seg_pos={0 0}, upload=14870]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:01:19.042699029 (14h21m18.632044017s) [upload=14870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:01:19.042703128 (14h21m18.632048115s) [upload=14870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:19.344335316 (14h21m18.933680293s) [upload=14870]: Done uploading segments... Aug 8 00:01:19.344353089 (14h21m18.933698077s) [upload=14870]: Upload finished. err: <nil> Aug 8 00:02:30.778225454 (14h22m30.367570441s) [upload=14892]: Starting upload Aug 8 00:02:30.778283114 (14h22m30.367628151s) [upload=14892]: Done starting upload Aug 8 00:02:30.778411841 (14h22m30.367756828s) [upload=14892]: Uploading segments... Aug 8 00:02:30.778539616 (14h22m30.367884623s) [upload=14892]: Got next segment. Inline: false Aug 8 00:02:30.778583389 (14h22m30.367928376s) [upload=14892]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:30.778587717 (14h22m30.367932704s) [upload=14892]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:03:30.135980490 (14h23m29.725325467s) [upload=14892]: Next segment err: context deadline exceeded Aug 8 00:03:30.136177137 (14h23m29.725522124s) [upload=14892]: Done uploading segments... Aug 8 00:03:30.136196834 (14h23m29.725541821s) [upload=14892]: Upload finished. err: context deadline exceeded Duration: 6.243405024s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X..........X.......X...X............X....X...............] duration: 6.153109943s FAIL (2025-08-08 00:50:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837172821 (14h59m54.426517798s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837385949 (14h59m54.426730936s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837507802 (14h59m54.426852779s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837640145 (14h59m54.426985122s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837819459 (14h59m54.427164446s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837962502 (14h59m54.427307479s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838112790 (14h59m54.427457767s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838238691 (14h59m54.427583678s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838393447 (14h59m54.427738434s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838490402 (14h59m54.427835389s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838660288 (14h59m54.428005275s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838681548 (14h59m54.428026525s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838866312 (14h59m54.428211299s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838893975 (14h59m54.428238962s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.839056986 (14h59m54.428401974s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:55.935384249 (14h59m55.524729246s) [upload=15536]: Next returned nil segment Aug 8 00:39:55.935402815 (14h59m55.524747802s) [upload=15536]: Waiting for error group managing segments... Aug 8 00:39:56.226788010 (14h59m55.816132997s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.265710491 (14h59m55.855055478s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.326363980 (14h59m55.915708967s) [noise=true , address=64.49.13.169:4709 , piece_id=PIPH24KH4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773039652 (14h59m56.362384679s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773213735 (14h59m56.362558732s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.782851711 (14h59m56.372196698s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854376586 (14h59m56.443721573s) [noise=true , address=104.36.82.46:4484 , piece_id=BMHSO2DIXVRRHUZRHWUQ24TXMMTRMVX34SVDUUUMRMKPSUQG3APQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854703231 (14h59m56.444048218s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.914818370 (14h59m56.504163357s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.915843802 (14h59m56.505188849s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.942005621 (14h59m56.531350608s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.981103286 (14h59m56.570448273s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.095564053 (14h59m56.68490904s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216182613 (14h59m56.8055276s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216206869 (14h59m56.805551856s) [seg_pos={0 0}, upload=15536]: Segment reached optimal threshold of 14 pieces. Aug 8 00:39:57.216406632 (14h59m56.805751619s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bN25eEEjxCWKJSr8qj7moDJxArgfsjzPTdBcLdfvoeyxiqPpc): protocol: expected piece hash; context canceled; EOF Aug 8 00:39:57.216460465 (14h59m56.805805452s) [noise=true , address=104.36.82.5:4869 , piece_id=JBCDCG676BBV6U2GAVWDQIWYXKKQGW2DKK6MDY74LNPGP4K3CTQA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection Aug 8 00:39:57.216644467 (14h59m56.805989454s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD): protocol: read tcp 188.166.225.170:50728->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216665567 (14h59m56.806010554s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: read tcp 188.166.225.170:45052->104.36.82.61:4383: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216679534 (14h59m56.806024521s) [seg_pos={0 0}, upload=15536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:39:57.216739699 (14h59m56.806084686s) [upload=15536]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:39:57.216743286 (14h59m56.806088273s) [upload=15536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:39:57.516798834 (14h59m57.106143821s) [upload=15536]: Done uploading segments... Aug 8 00:39:57.516817129 (14h59m57.106162116s) [upload=15536]: Upload finished. err: <nil> Aug 8 00:40:09.579830255 (15h0m9.169175242s) [upload=15541]: Starting upload Aug 8 00:40:09.579877025 (15h0m9.169222012s) [upload=15541]: Done starting upload Aug 8 00:40:09.579941739 (15h0m9.169286726s) [upload=15541]: Uploading segments... Aug 8 00:40:09.580164405 (15h0m9.169509432s) [upload=15541]: Got next segment. Inline: false Aug 8 00:40:09.580219460 (15h0m9.169564477s) [upload=15541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:40:09.580251472 (15h0m9.169596489s) [upload=15541]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:40:09.797368422 (15h0m9.386713409s) [upload=15541]: Next segment err: metaclient: Too Many Requests Aug 8 00:40:09.797398149 (15h0m9.386743136s) [upload=15541]: Done uploading segments... Aug 8 00:40:09.797473614 (15h0m9.386818601s) [upload=15541]: Upload finished. err: metaclient: Too Many Requests Duration: 6.153135834s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [X....X......................................................] duration: 1.860428608s FAIL (2025-08-08 00:50:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9109253s) [upload=14753]: Starting upload Aug 7 23:54:04.489828037 (14h14m4.079173014s) [upload=14753]: Done starting upload Aug 7 23:54:04.489877612 (14h14m4.079222589s) [upload=14753]: Uploading segments... Aug 7 23:54:04.489903933 (14h14m4.07924892s) [upload=14753]: Got next segment. Inline: true Aug 7 23:54:04.489907469 (14h14m4.079252446s) [upload=14753]: Waiting for error group managing segments... Aug 7 23:54:04.489939651 (14h14m4.079284638s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:04.489943298 (14h14m4.079288285s) [upload=14753]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:04.489945472 (14h14m4.079290459s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:04.778726062 (14h14m4.368071049s) [upload=14753]: Done uploading segments... Aug 7 23:54:04.778744989 (14h14m4.368089976s) [upload=14753]: Upload finished. err: <nil> Aug 7 23:55:14.199719855 (14h15m13.789064892s) [upload=14770]: Starting upload Aug 7 23:55:14.199777645 (14h15m13.789122632s) [upload=14770]: Done starting upload Aug 7 23:55:14.199824415 (14h15m13.789169392s) [upload=14770]: Uploading segments... Aug 7 23:55:14.199852819 (14h15m13.789197806s) [upload=14770]: Got next segment. Inline: true Aug 7 23:55:14.199856436 (14h15m13.789201423s) [upload=14770]: Waiting for error group managing segments... Aug 7 23:55:14.199889419 (14h15m13.789234406s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:14.199893557 (14h15m13.789238544s) [upload=14770]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:14.199895951 (14h15m13.789240938s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:13.542516296 (14h16m13.131861283s) [upload=14770]: Done uploading segments... Aug 7 23:56:13.542536334 (14h16m13.131881321s) [upload=14770]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:14.428677736 (14h16m14.018022723s) [upload=14788]: Starting upload Aug 7 23:56:14.428730978 (14h16m14.018075965s) [upload=14788]: Done starting upload Aug 7 23:56:14.428789289 (14h16m14.018134276s) [upload=14788]: Uploading segments... Aug 7 23:56:14.428819026 (14h16m14.018164013s) [upload=14788]: Got next segment. Inline: true Aug 7 23:56:14.428822873 (14h16m14.01816785s) [upload=14788]: Waiting for error group managing segments... Aug 7 23:56:14.428855215 (14h16m14.018200202s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:14.428858562 (14h16m14.018203549s) [upload=14788]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:14.428860736 (14h16m14.018205723s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:13.781984834 (14h17m13.371329821s) [upload=14788]: Done uploading segments... Aug 7 23:57:13.782003370 (14h17m13.371348357s) [upload=14788]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:29.079359478 (14h17m28.668704466s) [upload=14808]: Starting upload Aug 7 23:57:29.079416147 (14h17m28.668761134s) [upload=14808]: Done starting upload Aug 7 23:57:29.079461845 (14h17m28.668806822s) [upload=14808]: Uploading segments... Aug 7 23:57:29.079488876 (14h17m28.668833863s) [upload=14808]: Got next segment. Inline: true Aug 7 23:57:29.079492263 (14h17m28.66883725s) [upload=14808]: Waiting for error group managing segments... Aug 7 23:57:29.079523392 (14h17m28.668868379s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:29.079527049 (14h17m28.668872036s) [upload=14808]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:29.079529484 (14h17m28.668874461s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:28.439377526 (14h18m28.028722513s) [upload=14808]: Done uploading segments... Aug 7 23:58:28.439395721 (14h18m28.028740708s) [upload=14808]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:29.333669502 (14h18m28.923014489s) [upload=14825]: Starting upload Aug 7 23:58:29.333742581 (14h18m28.923087568s) [upload=14825]: Done starting upload Aug 7 23:58:29.333835740 (14h18m28.923180727s) [upload=14825]: Uploading segments... Aug 7 23:58:29.333878000 (14h18m28.923222987s) [upload=14825]: Got next segment. Inline: true Aug 7 23:58:29.333881998 (14h18m28.923226975s) [upload=14825]: Waiting for error group managing segments... Aug 7 23:58:29.333911455 (14h18m28.923256442s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:29.333917496 (14h18m28.923262473s) [upload=14825]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:29.333919741 (14h18m28.923264728s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:28.692415040 (14h19m28.281760027s) [upload=14825]: Done uploading segments... Aug 7 23:59:28.692444777 (14h19m28.281789764s) [upload=14825]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:29.562470053 (14h19m29.15181504s) [upload=14840]: Starting upload Aug 7 23:59:29.562538985 (14h19m29.151883972s) [upload=14840]: Done starting upload Aug 7 23:59:29.562614629 (14h19m29.151959616s) [upload=14840]: Uploading segments... Aug 7 23:59:29.562650589 (14h19m29.151995566s) [upload=14840]: Got next segment. Inline: true Aug 7 23:59:29.562655498 (14h19m29.152000485s) [upload=14840]: Waiting for error group managing segments... Aug 7 23:59:29.562686507 (14h19m29.152031494s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:29.562690264 (14h19m29.152035241s) [upload=14840]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:29.562692428 (14h19m29.152037405s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:29.822497550 (14h19m29.411842537s) [upload=14840]: Done uploading segments... Aug 7 23:59:29.822514693 (14h19m29.41185968s) [upload=14840]: Upload finished. err: <nil> Aug 8 00:00:35.820554363 (14h20m35.40989935s) [upload=14856]: Starting upload Aug 8 00:00:35.820610611 (14h20m35.409955598s) [upload=14856]: Done starting upload Aug 8 00:00:35.820669503 (14h20m35.4100145s) [upload=14856]: Uploading segments... Aug 8 00:00:35.820742333 (14h20m35.41008732s) [upload=14856]: Got next segment. Inline: true Aug 8 00:00:35.820747282 (14h20m35.410092269s) [upload=14856]: Waiting for error group managing segments... Aug 8 00:00:35.820788371 (14h20m35.410133358s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.820793050 (14h20m35.410138037s) [upload=14856]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.820795485 (14h20m35.410140472s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:36.084321816 (14h20m35.673666803s) [upload=14856]: Done uploading segments... Aug 8 00:00:36.084338618 (14h20m35.673683615s) [upload=14856]: Upload finished. err: <nil> Aug 8 00:01:14.496466774 (14h21m14.085811781s) [upload=14868]: Starting upload Aug 8 00:01:14.496556174 (14h21m14.085901162s) [upload=14868]: Done starting upload Aug 8 00:01:14.496609206 (14h21m14.085954183s) [upload=14868]: Uploading segments... Aug 8 00:01:14.496639985 (14h21m14.085984972s) [upload=14868]: Got next segment. Inline: true Aug 8 00:01:14.496643592 (14h21m14.085988579s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:01:14.496678638 (14h21m14.086023625s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:14.496682386 (14h21m14.086027373s) [upload=14868]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:14.496684681 (14h21m14.086029658s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:14.760681029 (14h21m14.350026016s) [upload=14868]: Done uploading segments... Aug 8 00:01:14.760702580 (14h21m14.350047567s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:01:50.577049881 (14h21m50.166394858s) [upload=14878]: Starting upload Aug 8 00:01:50.577100538 (14h21m50.166445525s) [upload=14878]: Done starting upload Aug 8 00:01:50.577180091 (14h21m50.166525078s) [upload=14878]: Uploading segments... Aug 8 00:01:50.577228012 (14h21m50.166573s) [upload=14878]: Got next segment. Inline: true Aug 8 00:01:50.577231469 (14h21m50.166576456s) [upload=14878]: Waiting for error group managing segments... Aug 8 00:01:50.577270633 (14h21m50.166615621s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:50.577274721 (14h21m50.166619708s) [upload=14878]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:50.577276906 (14h21m50.166621893s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:50.863237339 (14h21m50.452582326s) [upload=14878]: Done uploading segments... Aug 8 00:01:50.863254973 (14h21m50.452599961s) [upload=14878]: Upload finished. err: <nil> Aug 8 00:02:41.474648957 (14h22m41.063993944s) [upload=14899]: Starting upload Aug 8 00:02:41.474716297 (14h22m41.064061274s) [upload=14899]: Done starting upload Aug 8 00:02:41.474771743 (14h22m41.06411673s) [upload=14899]: Uploading segments... Aug 8 00:02:41.474801099 (14h22m41.064146086s) [upload=14899]: Got next segment. Inline: true Aug 8 00:02:41.474804966 (14h22m41.064149953s) [upload=14899]: Waiting for error group managing segments... Aug 8 00:02:41.474841336 (14h22m41.064186323s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:41.474845214 (14h22m41.064190201s) [upload=14899]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:41.474847558 (14h22m41.064192535s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:03:40.835731182 (14h23m40.425076189s) [upload=14899]: Done uploading segments... Aug 8 00:03:40.835778212 (14h23m40.425123199s) [upload=14899]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.860456331s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.835144283s FAIL (2025-08-08 00:50:07 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== =14964]: Done uploading segments... Aug 8 00:06:41.669168196 (14h26m41.258513183s) [upload=14964]: Upload finished. err: <nil> Aug 8 00:07:06.258431851 (14h27m5.847776838s) [upload=14976]: Starting upload Aug 8 00:07:06.258508298 (14h27m5.847853285s) [upload=14976]: Done starting upload Aug 8 00:07:06.258574986 (14h27m5.847919983s) [upload=14976]: Uploading segments... Aug 8 00:07:06.258618599 (14h27m5.847963616s) [upload=14976]: Got next segment. Inline: true Aug 8 00:07:06.258624600 (14h27m5.847969587s) [upload=14976]: Waiting for error group managing segments... Aug 8 00:07:06.258705555 (14h27m5.848050542s) [upload=14976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:07:06.258712268 (14h27m5.848057296s) [upload=14976]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:07:06.258716226 (14h27m5.848061253s) [upload=14976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:07:06.522670976 (14h27m6.112015963s) [upload=14976]: Done uploading segments... Aug 8 00:07:06.522717766 (14h27m6.112062773s) [upload=14976]: Upload finished. err: <nil> Aug 8 00:07:36.371369594 (14h27m35.960714581s) [upload=14980]: Starting upload Aug 8 00:07:36.371417074 (14h27m35.960762061s) [upload=14980]: Done starting upload Aug 8 00:07:36.371466890 (14h27m35.960811877s) [upload=14980]: Uploading segments... Aug 8 00:07:36.371508951 (14h27m35.960853938s) [upload=14980]: Got next segment. Inline: true Aug 8 00:07:36.371512888 (14h27m35.960857875s) [upload=14980]: Waiting for error group managing segments... Aug 8 00:07:36.371543446 (14h27m35.960888433s) [upload=14980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:07:36.371547434 (14h27m35.960892421s) [upload=14980]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:07:36.371549708 (14h27m35.960894695s) [upload=14980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:07:36.789946570 (14h27m36.379291557s) [upload=14980]: Done uploading segments... Aug 8 00:07:36.789964876 (14h27m36.379309863s) [upload=14980]: Upload finished. err: <nil> Aug 8 00:08:16.866388064 (14h28m16.455733051s) [upload=14994]: Starting upload Aug 8 00:08:16.866430886 (14h28m16.455775873s) [upload=14994]: Done starting upload Aug 8 00:08:16.866496882 (14h28m16.455841869s) [upload=14994]: Uploading segments... Aug 8 00:08:16.866525727 (14h28m16.455870714s) [upload=14994]: Got next segment. Inline: true Aug 8 00:08:16.866529494 (14h28m16.455874481s) [upload=14994]: Waiting for error group managing segments... Aug 8 00:08:16.866570884 (14h28m16.455915871s) [upload=14994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:08:16.866575212 (14h28m16.455920199s) [upload=14994]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:08:16.866577626 (14h28m16.455922613s) [upload=14994]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:08:17.138322091 (14h28m16.727667078s) [upload=14994]: Done uploading segments... Aug 8 00:08:17.138338402 (14h28m16.727683389s) [upload=14994]: Upload finished. err: <nil> Aug 8 00:08:57.633625328 (14h28m57.222970315s) [upload=15006]: Starting upload Aug 8 00:08:57.633689390 (14h28m57.223034378s) [upload=15006]: Done starting upload Aug 8 00:08:57.633744837 (14h28m57.223089824s) [upload=15006]: Uploading segments... Aug 8 00:08:57.633775525 (14h28m57.223120512s) [upload=15006]: Got next segment. Inline: true Aug 8 00:08:57.633779222 (14h28m57.223124199s) [upload=15006]: Waiting for error group managing segments... Aug 8 00:08:57.633951743 (14h28m57.22329672s) [upload=15006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:08:57.633956161 (14h28m57.223301148s) [upload=15006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:08:57.633958886 (14h28m57.223303873s) [upload=15006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:08:57.920842105 (14h28m57.510187102s) [upload=15006]: Done uploading segments... Aug 8 00:08:57.920865330 (14h28m57.510210327s) [upload=15006]: Upload finished. err: <nil> Aug 8 00:09:43.341371442 (14h29m42.930716449s) [upload=15015]: Starting upload Aug 8 00:09:43.341457767 (14h29m42.930802774s) [upload=15015]: Done starting upload Aug 8 00:09:43.341511951 (14h29m42.930856938s) [upload=15015]: Uploading segments... Aug 8 00:09:43.341579249 (14h29m42.930924236s) [upload=15015]: Got next segment. Inline: true Aug 8 00:09:43.341585642 (14h29m42.930930639s) [upload=15015]: Waiting for error group managing segments... Aug 8 00:09:43.341640437 (14h29m42.930985454s) [upload=15015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:09:43.341648422 (14h29m42.930993469s) [upload=15015]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:09:43.341653391 (14h29m42.930998408s) [upload=15015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:09:43.627931533 (14h29m43.21727652s) [upload=15015]: Done uploading segments... Aug 8 00:09:43.627948425 (14h29m43.217293402s) [upload=15015]: Upload finished. err: <nil> Aug 8 00:10:45.166177580 (14h30m44.755522568s) [upload=15031]: Starting upload Aug 8 00:10:45.166220523 (14h30m44.75556551s) [upload=15031]: Done starting upload Aug 8 00:10:45.166273034 (14h30m44.755618011s) [upload=15031]: Uploading segments... Aug 8 00:10:45.166297991 (14h30m44.755642979s) [upload=15031]: Got next segment. Inline: true Aug 8 00:10:45.166301568 (14h30m44.755646555s) [upload=15031]: Waiting for error group managing segments... Aug 8 00:10:45.166332277 (14h30m44.755677264s) [upload=15031]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:10:45.166335783 (14h30m44.75568077s) [upload=15031]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:10:45.166337998 (14h30m44.755682985s) [upload=15031]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:10:45.444515471 (14h30m45.033860458s) [upload=15031]: Done uploading segments... Aug 8 00:10:45.444533305 (14h30m45.033878282s) [upload=15031]: Upload finished. err: <nil> Aug 8 00:11:54.979425886 (14h31m54.568770863s) [upload=15052]: Starting upload Aug 8 00:11:54.979471082 (14h31m54.568816069s) [upload=15052]: Done starting upload Aug 8 00:11:54.979517260 (14h31m54.568862237s) [upload=15052]: Uploading segments... Aug 8 00:11:54.979549422 (14h31m54.568894399s) [upload=15052]: Got next segment. Inline: true Aug 8 00:11:54.979553008 (14h31m54.568897995s) [upload=15052]: Waiting for error group managing segments... Aug 8 00:11:54.979584328 (14h31m54.568929315s) [upload=15052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:11:54.979588356 (14h31m54.568933333s) [upload=15052]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:11:54.979590661 (14h31m54.568935648s) [upload=15052]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:11:55.340902393 (14h31m54.93024738s) [upload=15052]: Done uploading segments... Aug 8 00:11:55.340920908 (14h31m54.930265895s) [upload=15052]: Upload finished. err: <nil> Aug 8 00:12:44.503574128 (14h32m44.092919115s) [upload=15065]: Starting upload Aug 8 00:12:44.503634083 (14h32m44.09297907s) [upload=15065]: Done starting upload Aug 8 00:12:44.503710068 (14h32m44.093055055s) [upload=15065]: Uploading segments... Aug 8 00:12:44.503762669 (14h32m44.093107656s) [upload=15065]: Got next segment. Inline: true Aug 8 00:12:44.503767698 (14h32m44.093112685s) [upload=15065]: Waiting for error group managing segments... Aug 8 00:12:44.503805531 (14h32m44.093150508s) [upload=15065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:12:44.503809379 (14h32m44.093154366s) [upload=15065]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:12:44.503811633 (14h32m44.09315662s) [upload=15065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:12:44.792734473 (14h32m44.38207946s) [upload=15065]: Done uploading segments... Aug 8 00:12:44.792752288 (14h32m44.382097275s) [upload=15065]: Upload finished. err: <nil> Aug 8 00:13:19.961146705 (14h33m19.550491693s) [upload=15079]: Starting upload Aug 8 00:13:19.961187835 (14h33m19.550532822s) [upload=15079]: Done starting upload Aug 8 00:13:19.961236618 (14h33m19.550581605s) [upload=15079]: Uploading segments... Aug 8 00:13:19.961268168 (14h33m19.550613145s) [upload=15079]: Got next segment. Inline: true Aug 8 00:13:19.961273849 (14h33m19.550618836s) [upload=15079]: Waiting for error group managing segments... Aug 8 00:13:19.961304548 (14h33m19.550649525s) [upload=15079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:13:19.961308345 (14h33m19.550653332s) [upload=15079]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:13:19.961310910 (14h33m19.550655887s) [upload=15079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:13:20.385237160 (14h33m19.974582137s) [upload=15079]: Done uploading segments... Aug 8 00:13:20.385255485 (14h33m19.974600472s) [upload=15079]: Upload finished. err: <nil> Aug 8 00:14:06.785741109 (14h34m6.375086096s) [upload=15091]: Starting upload Aug 8 00:14:06.785899151 (14h34m6.375244138s) [upload=15091]: Done starting upload Aug 8 00:14:06.786002628 (14h34m6.375347615s) [upload=15091]: Uploading segments... Aug 8 00:14:06.786034078 (14h34m6.375379065s) [upload=15091]: Got next segment. Inline: true Aug 8 00:14:06.786037786 (14h34m6.375382773s) [upload=15091]: Waiting for error group managing segments... Aug 8 00:14:06.786073094 (14h34m6.375418081s) [upload=15091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:14:06.786078945 (14h34m6.375423922s) [upload=15091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:14:06.786081339 (14h34m6.375426326s) [upload=15091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:14:07.092788097 (14h34m6.682133084s) [upload=15091]: Done uploading segments... Aug 8 00:14:07.092803166 (14h34m6.682148153s) [upload=15091]: Upload finished. err: <nil> Duration: 1.835208655s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [X...X..........X.......X...X............X....X..............] duration: 6.051286032s FAIL (2025-08-08 00:50:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837172821 (14h59m54.426517798s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837385949 (14h59m54.426730936s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837507802 (14h59m54.426852779s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837640145 (14h59m54.426985122s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837819459 (14h59m54.427164446s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837962502 (14h59m54.427307479s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838112790 (14h59m54.427457767s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838238691 (14h59m54.427583678s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838393447 (14h59m54.427738434s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838490402 (14h59m54.427835389s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838660288 (14h59m54.428005275s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838681548 (14h59m54.428026525s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838866312 (14h59m54.428211299s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838893975 (14h59m54.428238962s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.839056986 (14h59m54.428401974s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:55.935384249 (14h59m55.524729246s) [upload=15536]: Next returned nil segment Aug 8 00:39:55.935402815 (14h59m55.524747802s) [upload=15536]: Waiting for error group managing segments... Aug 8 00:39:56.226788010 (14h59m55.816132997s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.265710491 (14h59m55.855055478s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.326363980 (14h59m55.915708967s) [noise=true , address=64.49.13.169:4709 , piece_id=PIPH24KH4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773039652 (14h59m56.362384679s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773213735 (14h59m56.362558732s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.782851711 (14h59m56.372196698s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854376586 (14h59m56.443721573s) [noise=true , address=104.36.82.46:4484 , piece_id=BMHSO2DIXVRRHUZRHWUQ24TXMMTRMVX34SVDUUUMRMKPSUQG3APQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854703231 (14h59m56.444048218s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.914818370 (14h59m56.504163357s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.915843802 (14h59m56.505188849s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.942005621 (14h59m56.531350608s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.981103286 (14h59m56.570448273s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.095564053 (14h59m56.68490904s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216182613 (14h59m56.8055276s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216206869 (14h59m56.805551856s) [seg_pos={0 0}, upload=15536]: Segment reached optimal threshold of 14 pieces. Aug 8 00:39:57.216406632 (14h59m56.805751619s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bN25eEEjxCWKJSr8qj7moDJxArgfsjzPTdBcLdfvoeyxiqPpc): protocol: expected piece hash; context canceled; EOF Aug 8 00:39:57.216460465 (14h59m56.805805452s) [noise=true , address=104.36.82.5:4869 , piece_id=JBCDCG676BBV6U2GAVWDQIWYXKKQGW2DKK6MDY74LNPGP4K3CTQA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection Aug 8 00:39:57.216644467 (14h59m56.805989454s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD): protocol: read tcp 188.166.225.170:50728->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216665567 (14h59m56.806010554s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: read tcp 188.166.225.170:45052->104.36.82.61:4383: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216679534 (14h59m56.806024521s) [seg_pos={0 0}, upload=15536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:39:57.216739699 (14h59m56.806084686s) [upload=15536]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:39:57.216743286 (14h59m56.806088273s) [upload=15536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:39:57.516798834 (14h59m57.106143821s) [upload=15536]: Done uploading segments... Aug 8 00:39:57.516817129 (14h59m57.106162116s) [upload=15536]: Upload finished. err: <nil> Aug 8 00:40:09.579830255 (15h0m9.169175242s) [upload=15541]: Starting upload Aug 8 00:40:09.579877025 (15h0m9.169222012s) [upload=15541]: Done starting upload Aug 8 00:40:09.579941739 (15h0m9.169286726s) [upload=15541]: Uploading segments... Aug 8 00:40:09.580164405 (15h0m9.169509432s) [upload=15541]: Got next segment. Inline: false Aug 8 00:40:09.580219460 (15h0m9.169564477s) [upload=15541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:40:09.580251472 (15h0m9.169596489s) [upload=15541]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:40:09.797368422 (15h0m9.386713409s) [upload=15541]: Next segment err: metaclient: Too Many Requests Aug 8 00:40:09.797398149 (15h0m9.386743136s) [upload=15541]: Done uploading segments... Aug 8 00:40:09.797473614 (15h0m9.386818601s) [upload=15541]: Upload finished. err: metaclient: Too Many Requests Duration: 6.051314215s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 3 out of 60 is failed (threshold=3) [XX....X.....................................................] duration: 1.837374981s FAIL (2025-08-08 00:49:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9109253s) [upload=14753]: Starting upload Aug 7 23:54:04.489828037 (14h14m4.079173014s) [upload=14753]: Done starting upload Aug 7 23:54:04.489877612 (14h14m4.079222589s) [upload=14753]: Uploading segments... Aug 7 23:54:04.489903933 (14h14m4.07924892s) [upload=14753]: Got next segment. Inline: true Aug 7 23:54:04.489907469 (14h14m4.079252446s) [upload=14753]: Waiting for error group managing segments... Aug 7 23:54:04.489939651 (14h14m4.079284638s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:04.489943298 (14h14m4.079288285s) [upload=14753]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:04.489945472 (14h14m4.079290459s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:04.778726062 (14h14m4.368071049s) [upload=14753]: Done uploading segments... Aug 7 23:54:04.778744989 (14h14m4.368089976s) [upload=14753]: Upload finished. err: <nil> Aug 7 23:55:14.199719855 (14h15m13.789064892s) [upload=14770]: Starting upload Aug 7 23:55:14.199777645 (14h15m13.789122632s) [upload=14770]: Done starting upload Aug 7 23:55:14.199824415 (14h15m13.789169392s) [upload=14770]: Uploading segments... Aug 7 23:55:14.199852819 (14h15m13.789197806s) [upload=14770]: Got next segment. Inline: true Aug 7 23:55:14.199856436 (14h15m13.789201423s) [upload=14770]: Waiting for error group managing segments... Aug 7 23:55:14.199889419 (14h15m13.789234406s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:14.199893557 (14h15m13.789238544s) [upload=14770]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:14.199895951 (14h15m13.789240938s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:13.542516296 (14h16m13.131861283s) [upload=14770]: Done uploading segments... Aug 7 23:56:13.542536334 (14h16m13.131881321s) [upload=14770]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:14.428677736 (14h16m14.018022723s) [upload=14788]: Starting upload Aug 7 23:56:14.428730978 (14h16m14.018075965s) [upload=14788]: Done starting upload Aug 7 23:56:14.428789289 (14h16m14.018134276s) [upload=14788]: Uploading segments... Aug 7 23:56:14.428819026 (14h16m14.018164013s) [upload=14788]: Got next segment. Inline: true Aug 7 23:56:14.428822873 (14h16m14.01816785s) [upload=14788]: Waiting for error group managing segments... Aug 7 23:56:14.428855215 (14h16m14.018200202s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:14.428858562 (14h16m14.018203549s) [upload=14788]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:14.428860736 (14h16m14.018205723s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:13.781984834 (14h17m13.371329821s) [upload=14788]: Done uploading segments... Aug 7 23:57:13.782003370 (14h17m13.371348357s) [upload=14788]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:29.079359478 (14h17m28.668704466s) [upload=14808]: Starting upload Aug 7 23:57:29.079416147 (14h17m28.668761134s) [upload=14808]: Done starting upload Aug 7 23:57:29.079461845 (14h17m28.668806822s) [upload=14808]: Uploading segments... Aug 7 23:57:29.079488876 (14h17m28.668833863s) [upload=14808]: Got next segment. Inline: true Aug 7 23:57:29.079492263 (14h17m28.66883725s) [upload=14808]: Waiting for error group managing segments... Aug 7 23:57:29.079523392 (14h17m28.668868379s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:29.079527049 (14h17m28.668872036s) [upload=14808]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:29.079529484 (14h17m28.668874461s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:28.439377526 (14h18m28.028722513s) [upload=14808]: Done uploading segments... Aug 7 23:58:28.439395721 (14h18m28.028740708s) [upload=14808]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:29.333669502 (14h18m28.923014489s) [upload=14825]: Starting upload Aug 7 23:58:29.333742581 (14h18m28.923087568s) [upload=14825]: Done starting upload Aug 7 23:58:29.333835740 (14h18m28.923180727s) [upload=14825]: Uploading segments... Aug 7 23:58:29.333878000 (14h18m28.923222987s) [upload=14825]: Got next segment. Inline: true Aug 7 23:58:29.333881998 (14h18m28.923226975s) [upload=14825]: Waiting for error group managing segments... Aug 7 23:58:29.333911455 (14h18m28.923256442s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:29.333917496 (14h18m28.923262473s) [upload=14825]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:29.333919741 (14h18m28.923264728s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:28.692415040 (14h19m28.281760027s) [upload=14825]: Done uploading segments... Aug 7 23:59:28.692444777 (14h19m28.281789764s) [upload=14825]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:29.562470053 (14h19m29.15181504s) [upload=14840]: Starting upload Aug 7 23:59:29.562538985 (14h19m29.151883972s) [upload=14840]: Done starting upload Aug 7 23:59:29.562614629 (14h19m29.151959616s) [upload=14840]: Uploading segments... Aug 7 23:59:29.562650589 (14h19m29.151995566s) [upload=14840]: Got next segment. Inline: true Aug 7 23:59:29.562655498 (14h19m29.152000485s) [upload=14840]: Waiting for error group managing segments... Aug 7 23:59:29.562686507 (14h19m29.152031494s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:29.562690264 (14h19m29.152035241s) [upload=14840]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:29.562692428 (14h19m29.152037405s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:29.822497550 (14h19m29.411842537s) [upload=14840]: Done uploading segments... Aug 7 23:59:29.822514693 (14h19m29.41185968s) [upload=14840]: Upload finished. err: <nil> Aug 8 00:00:35.820554363 (14h20m35.40989935s) [upload=14856]: Starting upload Aug 8 00:00:35.820610611 (14h20m35.409955598s) [upload=14856]: Done starting upload Aug 8 00:00:35.820669503 (14h20m35.4100145s) [upload=14856]: Uploading segments... Aug 8 00:00:35.820742333 (14h20m35.41008732s) [upload=14856]: Got next segment. Inline: true Aug 8 00:00:35.820747282 (14h20m35.410092269s) [upload=14856]: Waiting for error group managing segments... Aug 8 00:00:35.820788371 (14h20m35.410133358s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.820793050 (14h20m35.410138037s) [upload=14856]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.820795485 (14h20m35.410140472s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:36.084321816 (14h20m35.673666803s) [upload=14856]: Done uploading segments... Aug 8 00:00:36.084338618 (14h20m35.673683615s) [upload=14856]: Upload finished. err: <nil> Aug 8 00:01:14.496466774 (14h21m14.085811781s) [upload=14868]: Starting upload Aug 8 00:01:14.496556174 (14h21m14.085901162s) [upload=14868]: Done starting upload Aug 8 00:01:14.496609206 (14h21m14.085954183s) [upload=14868]: Uploading segments... Aug 8 00:01:14.496639985 (14h21m14.085984972s) [upload=14868]: Got next segment. Inline: true Aug 8 00:01:14.496643592 (14h21m14.085988579s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:01:14.496678638 (14h21m14.086023625s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:14.496682386 (14h21m14.086027373s) [upload=14868]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:14.496684681 (14h21m14.086029658s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:14.760681029 (14h21m14.350026016s) [upload=14868]: Done uploading segments... Aug 8 00:01:14.760702580 (14h21m14.350047567s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:01:50.577049881 (14h21m50.166394858s) [upload=14878]: Starting upload Aug 8 00:01:50.577100538 (14h21m50.166445525s) [upload=14878]: Done starting upload Aug 8 00:01:50.577180091 (14h21m50.166525078s) [upload=14878]: Uploading segments... Aug 8 00:01:50.577228012 (14h21m50.166573s) [upload=14878]: Got next segment. Inline: true Aug 8 00:01:50.577231469 (14h21m50.166576456s) [upload=14878]: Waiting for error group managing segments... Aug 8 00:01:50.577270633 (14h21m50.166615621s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:50.577274721 (14h21m50.166619708s) [upload=14878]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:50.577276906 (14h21m50.166621893s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:50.863237339 (14h21m50.452582326s) [upload=14878]: Done uploading segments... Aug 8 00:01:50.863254973 (14h21m50.452599961s) [upload=14878]: Upload finished. err: <nil> Aug 8 00:02:41.474648957 (14h22m41.063993944s) [upload=14899]: Starting upload Aug 8 00:02:41.474716297 (14h22m41.064061274s) [upload=14899]: Done starting upload Aug 8 00:02:41.474771743 (14h22m41.06411673s) [upload=14899]: Uploading segments... Aug 8 00:02:41.474801099 (14h22m41.064146086s) [upload=14899]: Got next segment. Inline: true Aug 8 00:02:41.474804966 (14h22m41.064149953s) [upload=14899]: Waiting for error group managing segments... Aug 8 00:02:41.474841336 (14h22m41.064186323s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:41.474845214 (14h22m41.064190201s) [upload=14899]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:41.474847558 (14h22m41.064192535s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:03:40.835731182 (14h23m40.425076189s) [upload=14899]: Done uploading segments... Aug 8 00:03:40.835778212 (14h23m40.425123199s) [upload=14899]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.837402814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.441552719s FAIL (2025-08-08 00:49:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.246137011 (14h21m15.835481998s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.246560380 (14h21m15.835905367s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247408974 (14h21m15.836753961s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247678410 (14h21m15.837023397s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247919882 (14h21m15.837264869s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247943888 (14h21m15.837288875s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248293266 (14h21m15.837638253s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248420540 (14h21m15.837765527s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248557311 (14h21m15.837902288s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248758506 (14h21m15.838103493s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248897792 (14h21m15.838242779s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248989629 (14h21m15.838334616s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249186575 (14h21m15.838531552s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249273772 (14h21m15.838618759s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249459598 (14h21m15.838804585s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249717240 (14h21m15.839062227s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:17.413257645 (14h21m17.002602632s) [upload=14870]: Next returned nil segment Aug 8 00:01:17.413273324 (14h21m17.002618311s) [upload=14870]: Waiting for error group managing segments... Aug 8 00:01:17.710897294 (14h21m17.300242281s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:17.715058896 (14h21m17.304403883s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.126218161 (14h21m17.715563148s) [noise=true , address=79.127.201.215:4356 , piece_id=VWNEIHNI4KPPF4QLW36INIWTUI2CD6MZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.139272367 (14h21m17.728617354s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.221269031 (14h21m17.810614018s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.257350144 (14h21m17.846695121s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.258435089 (14h21m17.847780076s) [noise=true , address=104.36.82.44:4876 , piece_id=BA2G42OXQD3NLONUVD475S2BNC7T37T5PHPE3K3E5HJAJP4W5MKQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.287010923 (14h21m17.87635591s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.348793001 (14h21m17.938137988s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.378129459 (14h21m17.967474436s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.386414901 (14h21m17.975759888s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.516150823 (14h21m18.10549581s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.626647204 (14h21m18.215992191s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:19.042108811 (14h21m18.631453798s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:19.042136654 (14h21m18.631481641s) [seg_pos={0 0}, upload=14870]: Segment reached optimal threshold of 14 pieces. Aug 8 00:01:19.042375541 (14h21m18.631720518s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:19.042485912 (14h21m18.6318309s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: rpc: dial tcp 104.36.82.55:4209: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:01:19.042532361 (14h21m18.631877348s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqNLPRWY7fJRS6F9h4B4qVFv6U5jyWkCy3Um7jfjEdCShEjsD): protocol: write tcp 188.166.225.170:37056->104.36.82.20:4466: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:01:19.042616462 (14h21m18.631961449s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa): protocol: rpc: dial tcp 104.36.82.5:4844: operation was canceled; conn closed: op; piecestore: piecestore close: context canceled Aug 8 00:01:19.042628876 (14h21m18.631973863s) [seg_pos={0 0}, upload=14870]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:01:19.042699029 (14h21m18.632044017s) [upload=14870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:01:19.042703128 (14h21m18.632048115s) [upload=14870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:19.344335316 (14h21m18.933680293s) [upload=14870]: Done uploading segments... Aug 8 00:01:19.344353089 (14h21m18.933698077s) [upload=14870]: Upload finished. err: <nil> Aug 8 00:02:30.778225454 (14h22m30.367570441s) [upload=14892]: Starting upload Aug 8 00:02:30.778283114 (14h22m30.367628151s) [upload=14892]: Done starting upload Aug 8 00:02:30.778411841 (14h22m30.367756828s) [upload=14892]: Uploading segments... Aug 8 00:02:30.778539616 (14h22m30.367884623s) [upload=14892]: Got next segment. Inline: false Aug 8 00:02:30.778583389 (14h22m30.367928376s) [upload=14892]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:30.778587717 (14h22m30.367932704s) [upload=14892]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:03:30.135980490 (14h23m29.725325467s) [upload=14892]: Next segment err: context deadline exceeded Aug 8 00:03:30.136177137 (14h23m29.725522124s) [upload=14892]: Done uploading segments... Aug 8 00:03:30.136196834 (14h23m29.725541821s) [upload=14892]: Upload finished. err: context deadline exceeded Duration: 6.441580492s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X...X..........X.......X...X............X....X.............] duration: 6.426353471s FAIL (2025-08-08 00:49:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837172821 (14h59m54.426517798s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837385949 (14h59m54.426730936s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837507802 (14h59m54.426852779s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837640145 (14h59m54.426985122s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837819459 (14h59m54.427164446s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837962502 (14h59m54.427307479s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838112790 (14h59m54.427457767s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838238691 (14h59m54.427583678s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838393447 (14h59m54.427738434s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838490402 (14h59m54.427835389s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838660288 (14h59m54.428005275s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838681548 (14h59m54.428026525s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838866312 (14h59m54.428211299s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838893975 (14h59m54.428238962s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.839056986 (14h59m54.428401974s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:55.935384249 (14h59m55.524729246s) [upload=15536]: Next returned nil segment Aug 8 00:39:55.935402815 (14h59m55.524747802s) [upload=15536]: Waiting for error group managing segments... Aug 8 00:39:56.226788010 (14h59m55.816132997s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.265710491 (14h59m55.855055478s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.326363980 (14h59m55.915708967s) [noise=true , address=64.49.13.169:4709 , piece_id=PIPH24KH4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773039652 (14h59m56.362384679s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773213735 (14h59m56.362558732s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.782851711 (14h59m56.372196698s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854376586 (14h59m56.443721573s) [noise=true , address=104.36.82.46:4484 , piece_id=BMHSO2DIXVRRHUZRHWUQ24TXMMTRMVX34SVDUUUMRMKPSUQG3APQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854703231 (14h59m56.444048218s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.914818370 (14h59m56.504163357s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.915843802 (14h59m56.505188849s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.942005621 (14h59m56.531350608s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.981103286 (14h59m56.570448273s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.095564053 (14h59m56.68490904s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216182613 (14h59m56.8055276s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216206869 (14h59m56.805551856s) [seg_pos={0 0}, upload=15536]: Segment reached optimal threshold of 14 pieces. Aug 8 00:39:57.216406632 (14h59m56.805751619s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bN25eEEjxCWKJSr8qj7moDJxArgfsjzPTdBcLdfvoeyxiqPpc): protocol: expected piece hash; context canceled; EOF Aug 8 00:39:57.216460465 (14h59m56.805805452s) [noise=true , address=104.36.82.5:4869 , piece_id=JBCDCG676BBV6U2GAVWDQIWYXKKQGW2DKK6MDY74LNPGP4K3CTQA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection Aug 8 00:39:57.216644467 (14h59m56.805989454s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD): protocol: read tcp 188.166.225.170:50728->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216665567 (14h59m56.806010554s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: read tcp 188.166.225.170:45052->104.36.82.61:4383: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216679534 (14h59m56.806024521s) [seg_pos={0 0}, upload=15536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:39:57.216739699 (14h59m56.806084686s) [upload=15536]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:39:57.216743286 (14h59m56.806088273s) [upload=15536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:39:57.516798834 (14h59m57.106143821s) [upload=15536]: Done uploading segments... Aug 8 00:39:57.516817129 (14h59m57.106162116s) [upload=15536]: Upload finished. err: <nil> Aug 8 00:40:09.579830255 (15h0m9.169175242s) [upload=15541]: Starting upload Aug 8 00:40:09.579877025 (15h0m9.169222012s) [upload=15541]: Done starting upload Aug 8 00:40:09.579941739 (15h0m9.169286726s) [upload=15541]: Uploading segments... Aug 8 00:40:09.580164405 (15h0m9.169509432s) [upload=15541]: Got next segment. Inline: false Aug 8 00:40:09.580219460 (15h0m9.169564477s) [upload=15541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:40:09.580251472 (15h0m9.169596489s) [upload=15541]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:40:09.797368422 (15h0m9.386713409s) [upload=15541]: Next segment err: metaclient: Too Many Requests Aug 8 00:40:09.797398149 (15h0m9.386743136s) [upload=15541]: Done uploading segments... Aug 8 00:40:09.797473614 (15h0m9.386818601s) [upload=15541]: Upload finished. err: metaclient: Too Many Requests Duration: 6.426377225s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 4 out of 60 is failed (threshold=3) [XXX....X....................................................] duration: 1.884252331s FAIL (2025-08-08 00:49:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9109253s) [upload=14753]: Starting upload Aug 7 23:54:04.489828037 (14h14m4.079173014s) [upload=14753]: Done starting upload Aug 7 23:54:04.489877612 (14h14m4.079222589s) [upload=14753]: Uploading segments... Aug 7 23:54:04.489903933 (14h14m4.07924892s) [upload=14753]: Got next segment. Inline: true Aug 7 23:54:04.489907469 (14h14m4.079252446s) [upload=14753]: Waiting for error group managing segments... Aug 7 23:54:04.489939651 (14h14m4.079284638s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:04.489943298 (14h14m4.079288285s) [upload=14753]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:04.489945472 (14h14m4.079290459s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:04.778726062 (14h14m4.368071049s) [upload=14753]: Done uploading segments... Aug 7 23:54:04.778744989 (14h14m4.368089976s) [upload=14753]: Upload finished. err: <nil> Aug 7 23:55:14.199719855 (14h15m13.789064892s) [upload=14770]: Starting upload Aug 7 23:55:14.199777645 (14h15m13.789122632s) [upload=14770]: Done starting upload Aug 7 23:55:14.199824415 (14h15m13.789169392s) [upload=14770]: Uploading segments... Aug 7 23:55:14.199852819 (14h15m13.789197806s) [upload=14770]: Got next segment. Inline: true Aug 7 23:55:14.199856436 (14h15m13.789201423s) [upload=14770]: Waiting for error group managing segments... Aug 7 23:55:14.199889419 (14h15m13.789234406s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:14.199893557 (14h15m13.789238544s) [upload=14770]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:14.199895951 (14h15m13.789240938s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:13.542516296 (14h16m13.131861283s) [upload=14770]: Done uploading segments... Aug 7 23:56:13.542536334 (14h16m13.131881321s) [upload=14770]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:14.428677736 (14h16m14.018022723s) [upload=14788]: Starting upload Aug 7 23:56:14.428730978 (14h16m14.018075965s) [upload=14788]: Done starting upload Aug 7 23:56:14.428789289 (14h16m14.018134276s) [upload=14788]: Uploading segments... Aug 7 23:56:14.428819026 (14h16m14.018164013s) [upload=14788]: Got next segment. Inline: true Aug 7 23:56:14.428822873 (14h16m14.01816785s) [upload=14788]: Waiting for error group managing segments... Aug 7 23:56:14.428855215 (14h16m14.018200202s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:14.428858562 (14h16m14.018203549s) [upload=14788]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:14.428860736 (14h16m14.018205723s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:13.781984834 (14h17m13.371329821s) [upload=14788]: Done uploading segments... Aug 7 23:57:13.782003370 (14h17m13.371348357s) [upload=14788]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:29.079359478 (14h17m28.668704466s) [upload=14808]: Starting upload Aug 7 23:57:29.079416147 (14h17m28.668761134s) [upload=14808]: Done starting upload Aug 7 23:57:29.079461845 (14h17m28.668806822s) [upload=14808]: Uploading segments... Aug 7 23:57:29.079488876 (14h17m28.668833863s) [upload=14808]: Got next segment. Inline: true Aug 7 23:57:29.079492263 (14h17m28.66883725s) [upload=14808]: Waiting for error group managing segments... Aug 7 23:57:29.079523392 (14h17m28.668868379s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:29.079527049 (14h17m28.668872036s) [upload=14808]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:29.079529484 (14h17m28.668874461s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:28.439377526 (14h18m28.028722513s) [upload=14808]: Done uploading segments... Aug 7 23:58:28.439395721 (14h18m28.028740708s) [upload=14808]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:29.333669502 (14h18m28.923014489s) [upload=14825]: Starting upload Aug 7 23:58:29.333742581 (14h18m28.923087568s) [upload=14825]: Done starting upload Aug 7 23:58:29.333835740 (14h18m28.923180727s) [upload=14825]: Uploading segments... Aug 7 23:58:29.333878000 (14h18m28.923222987s) [upload=14825]: Got next segment. Inline: true Aug 7 23:58:29.333881998 (14h18m28.923226975s) [upload=14825]: Waiting for error group managing segments... Aug 7 23:58:29.333911455 (14h18m28.923256442s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:29.333917496 (14h18m28.923262473s) [upload=14825]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:29.333919741 (14h18m28.923264728s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:28.692415040 (14h19m28.281760027s) [upload=14825]: Done uploading segments... Aug 7 23:59:28.692444777 (14h19m28.281789764s) [upload=14825]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:29.562470053 (14h19m29.15181504s) [upload=14840]: Starting upload Aug 7 23:59:29.562538985 (14h19m29.151883972s) [upload=14840]: Done starting upload Aug 7 23:59:29.562614629 (14h19m29.151959616s) [upload=14840]: Uploading segments... Aug 7 23:59:29.562650589 (14h19m29.151995566s) [upload=14840]: Got next segment. Inline: true Aug 7 23:59:29.562655498 (14h19m29.152000485s) [upload=14840]: Waiting for error group managing segments... Aug 7 23:59:29.562686507 (14h19m29.152031494s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:29.562690264 (14h19m29.152035241s) [upload=14840]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:29.562692428 (14h19m29.152037405s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:29.822497550 (14h19m29.411842537s) [upload=14840]: Done uploading segments... Aug 7 23:59:29.822514693 (14h19m29.41185968s) [upload=14840]: Upload finished. err: <nil> Aug 8 00:00:35.820554363 (14h20m35.40989935s) [upload=14856]: Starting upload Aug 8 00:00:35.820610611 (14h20m35.409955598s) [upload=14856]: Done starting upload Aug 8 00:00:35.820669503 (14h20m35.4100145s) [upload=14856]: Uploading segments... Aug 8 00:00:35.820742333 (14h20m35.41008732s) [upload=14856]: Got next segment. Inline: true Aug 8 00:00:35.820747282 (14h20m35.410092269s) [upload=14856]: Waiting for error group managing segments... Aug 8 00:00:35.820788371 (14h20m35.410133358s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.820793050 (14h20m35.410138037s) [upload=14856]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.820795485 (14h20m35.410140472s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:36.084321816 (14h20m35.673666803s) [upload=14856]: Done uploading segments... Aug 8 00:00:36.084338618 (14h20m35.673683615s) [upload=14856]: Upload finished. err: <nil> Aug 8 00:01:14.496466774 (14h21m14.085811781s) [upload=14868]: Starting upload Aug 8 00:01:14.496556174 (14h21m14.085901162s) [upload=14868]: Done starting upload Aug 8 00:01:14.496609206 (14h21m14.085954183s) [upload=14868]: Uploading segments... Aug 8 00:01:14.496639985 (14h21m14.085984972s) [upload=14868]: Got next segment. Inline: true Aug 8 00:01:14.496643592 (14h21m14.085988579s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:01:14.496678638 (14h21m14.086023625s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:14.496682386 (14h21m14.086027373s) [upload=14868]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:14.496684681 (14h21m14.086029658s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:14.760681029 (14h21m14.350026016s) [upload=14868]: Done uploading segments... Aug 8 00:01:14.760702580 (14h21m14.350047567s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:01:50.577049881 (14h21m50.166394858s) [upload=14878]: Starting upload Aug 8 00:01:50.577100538 (14h21m50.166445525s) [upload=14878]: Done starting upload Aug 8 00:01:50.577180091 (14h21m50.166525078s) [upload=14878]: Uploading segments... Aug 8 00:01:50.577228012 (14h21m50.166573s) [upload=14878]: Got next segment. Inline: true Aug 8 00:01:50.577231469 (14h21m50.166576456s) [upload=14878]: Waiting for error group managing segments... Aug 8 00:01:50.577270633 (14h21m50.166615621s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:50.577274721 (14h21m50.166619708s) [upload=14878]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:50.577276906 (14h21m50.166621893s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:50.863237339 (14h21m50.452582326s) [upload=14878]: Done uploading segments... Aug 8 00:01:50.863254973 (14h21m50.452599961s) [upload=14878]: Upload finished. err: <nil> Aug 8 00:02:41.474648957 (14h22m41.063993944s) [upload=14899]: Starting upload Aug 8 00:02:41.474716297 (14h22m41.064061274s) [upload=14899]: Done starting upload Aug 8 00:02:41.474771743 (14h22m41.06411673s) [upload=14899]: Uploading segments... Aug 8 00:02:41.474801099 (14h22m41.064146086s) [upload=14899]: Got next segment. Inline: true Aug 8 00:02:41.474804966 (14h22m41.064149953s) [upload=14899]: Waiting for error group managing segments... Aug 8 00:02:41.474841336 (14h22m41.064186323s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:41.474845214 (14h22m41.064190201s) [upload=14899]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:41.474847558 (14h22m41.064192535s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:03:40.835731182 (14h23m40.425076189s) [upload=14899]: Done uploading segments... Aug 8 00:03:40.835778212 (14h23m40.425123199s) [upload=14899]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.884276237s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.852273824s FAIL (2025-08-08 00:49:02 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== =14964]: Done uploading segments... Aug 8 00:06:41.669168196 (14h26m41.258513183s) [upload=14964]: Upload finished. err: <nil> Aug 8 00:07:06.258431851 (14h27m5.847776838s) [upload=14976]: Starting upload Aug 8 00:07:06.258508298 (14h27m5.847853285s) [upload=14976]: Done starting upload Aug 8 00:07:06.258574986 (14h27m5.847919983s) [upload=14976]: Uploading segments... Aug 8 00:07:06.258618599 (14h27m5.847963616s) [upload=14976]: Got next segment. Inline: true Aug 8 00:07:06.258624600 (14h27m5.847969587s) [upload=14976]: Waiting for error group managing segments... Aug 8 00:07:06.258705555 (14h27m5.848050542s) [upload=14976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:07:06.258712268 (14h27m5.848057296s) [upload=14976]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:07:06.258716226 (14h27m5.848061253s) [upload=14976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:07:06.522670976 (14h27m6.112015963s) [upload=14976]: Done uploading segments... Aug 8 00:07:06.522717766 (14h27m6.112062773s) [upload=14976]: Upload finished. err: <nil> Aug 8 00:07:36.371369594 (14h27m35.960714581s) [upload=14980]: Starting upload Aug 8 00:07:36.371417074 (14h27m35.960762061s) [upload=14980]: Done starting upload Aug 8 00:07:36.371466890 (14h27m35.960811877s) [upload=14980]: Uploading segments... Aug 8 00:07:36.371508951 (14h27m35.960853938s) [upload=14980]: Got next segment. Inline: true Aug 8 00:07:36.371512888 (14h27m35.960857875s) [upload=14980]: Waiting for error group managing segments... Aug 8 00:07:36.371543446 (14h27m35.960888433s) [upload=14980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:07:36.371547434 (14h27m35.960892421s) [upload=14980]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:07:36.371549708 (14h27m35.960894695s) [upload=14980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:07:36.789946570 (14h27m36.379291557s) [upload=14980]: Done uploading segments... Aug 8 00:07:36.789964876 (14h27m36.379309863s) [upload=14980]: Upload finished. err: <nil> Aug 8 00:08:16.866388064 (14h28m16.455733051s) [upload=14994]: Starting upload Aug 8 00:08:16.866430886 (14h28m16.455775873s) [upload=14994]: Done starting upload Aug 8 00:08:16.866496882 (14h28m16.455841869s) [upload=14994]: Uploading segments... Aug 8 00:08:16.866525727 (14h28m16.455870714s) [upload=14994]: Got next segment. Inline: true Aug 8 00:08:16.866529494 (14h28m16.455874481s) [upload=14994]: Waiting for error group managing segments... Aug 8 00:08:16.866570884 (14h28m16.455915871s) [upload=14994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:08:16.866575212 (14h28m16.455920199s) [upload=14994]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:08:16.866577626 (14h28m16.455922613s) [upload=14994]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:08:17.138322091 (14h28m16.727667078s) [upload=14994]: Done uploading segments... Aug 8 00:08:17.138338402 (14h28m16.727683389s) [upload=14994]: Upload finished. err: <nil> Aug 8 00:08:57.633625328 (14h28m57.222970315s) [upload=15006]: Starting upload Aug 8 00:08:57.633689390 (14h28m57.223034378s) [upload=15006]: Done starting upload Aug 8 00:08:57.633744837 (14h28m57.223089824s) [upload=15006]: Uploading segments... Aug 8 00:08:57.633775525 (14h28m57.223120512s) [upload=15006]: Got next segment. Inline: true Aug 8 00:08:57.633779222 (14h28m57.223124199s) [upload=15006]: Waiting for error group managing segments... Aug 8 00:08:57.633951743 (14h28m57.22329672s) [upload=15006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:08:57.633956161 (14h28m57.223301148s) [upload=15006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:08:57.633958886 (14h28m57.223303873s) [upload=15006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:08:57.920842105 (14h28m57.510187102s) [upload=15006]: Done uploading segments... Aug 8 00:08:57.920865330 (14h28m57.510210327s) [upload=15006]: Upload finished. err: <nil> Aug 8 00:09:43.341371442 (14h29m42.930716449s) [upload=15015]: Starting upload Aug 8 00:09:43.341457767 (14h29m42.930802774s) [upload=15015]: Done starting upload Aug 8 00:09:43.341511951 (14h29m42.930856938s) [upload=15015]: Uploading segments... Aug 8 00:09:43.341579249 (14h29m42.930924236s) [upload=15015]: Got next segment. Inline: true Aug 8 00:09:43.341585642 (14h29m42.930930639s) [upload=15015]: Waiting for error group managing segments... Aug 8 00:09:43.341640437 (14h29m42.930985454s) [upload=15015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:09:43.341648422 (14h29m42.930993469s) [upload=15015]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:09:43.341653391 (14h29m42.930998408s) [upload=15015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:09:43.627931533 (14h29m43.21727652s) [upload=15015]: Done uploading segments... Aug 8 00:09:43.627948425 (14h29m43.217293402s) [upload=15015]: Upload finished. err: <nil> Aug 8 00:10:45.166177580 (14h30m44.755522568s) [upload=15031]: Starting upload Aug 8 00:10:45.166220523 (14h30m44.75556551s) [upload=15031]: Done starting upload Aug 8 00:10:45.166273034 (14h30m44.755618011s) [upload=15031]: Uploading segments... Aug 8 00:10:45.166297991 (14h30m44.755642979s) [upload=15031]: Got next segment. Inline: true Aug 8 00:10:45.166301568 (14h30m44.755646555s) [upload=15031]: Waiting for error group managing segments... Aug 8 00:10:45.166332277 (14h30m44.755677264s) [upload=15031]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:10:45.166335783 (14h30m44.75568077s) [upload=15031]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:10:45.166337998 (14h30m44.755682985s) [upload=15031]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:10:45.444515471 (14h30m45.033860458s) [upload=15031]: Done uploading segments... Aug 8 00:10:45.444533305 (14h30m45.033878282s) [upload=15031]: Upload finished. err: <nil> Aug 8 00:11:54.979425886 (14h31m54.568770863s) [upload=15052]: Starting upload Aug 8 00:11:54.979471082 (14h31m54.568816069s) [upload=15052]: Done starting upload Aug 8 00:11:54.979517260 (14h31m54.568862237s) [upload=15052]: Uploading segments... Aug 8 00:11:54.979549422 (14h31m54.568894399s) [upload=15052]: Got next segment. Inline: true Aug 8 00:11:54.979553008 (14h31m54.568897995s) [upload=15052]: Waiting for error group managing segments... Aug 8 00:11:54.979584328 (14h31m54.568929315s) [upload=15052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:11:54.979588356 (14h31m54.568933333s) [upload=15052]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:11:54.979590661 (14h31m54.568935648s) [upload=15052]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:11:55.340902393 (14h31m54.93024738s) [upload=15052]: Done uploading segments... Aug 8 00:11:55.340920908 (14h31m54.930265895s) [upload=15052]: Upload finished. err: <nil> Aug 8 00:12:44.503574128 (14h32m44.092919115s) [upload=15065]: Starting upload Aug 8 00:12:44.503634083 (14h32m44.09297907s) [upload=15065]: Done starting upload Aug 8 00:12:44.503710068 (14h32m44.093055055s) [upload=15065]: Uploading segments... Aug 8 00:12:44.503762669 (14h32m44.093107656s) [upload=15065]: Got next segment. Inline: true Aug 8 00:12:44.503767698 (14h32m44.093112685s) [upload=15065]: Waiting for error group managing segments... Aug 8 00:12:44.503805531 (14h32m44.093150508s) [upload=15065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:12:44.503809379 (14h32m44.093154366s) [upload=15065]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:12:44.503811633 (14h32m44.09315662s) [upload=15065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:12:44.792734473 (14h32m44.38207946s) [upload=15065]: Done uploading segments... Aug 8 00:12:44.792752288 (14h32m44.382097275s) [upload=15065]: Upload finished. err: <nil> Aug 8 00:13:19.961146705 (14h33m19.550491693s) [upload=15079]: Starting upload Aug 8 00:13:19.961187835 (14h33m19.550532822s) [upload=15079]: Done starting upload Aug 8 00:13:19.961236618 (14h33m19.550581605s) [upload=15079]: Uploading segments... Aug 8 00:13:19.961268168 (14h33m19.550613145s) [upload=15079]: Got next segment. Inline: true Aug 8 00:13:19.961273849 (14h33m19.550618836s) [upload=15079]: Waiting for error group managing segments... Aug 8 00:13:19.961304548 (14h33m19.550649525s) [upload=15079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:13:19.961308345 (14h33m19.550653332s) [upload=15079]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:13:19.961310910 (14h33m19.550655887s) [upload=15079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:13:20.385237160 (14h33m19.974582137s) [upload=15079]: Done uploading segments... Aug 8 00:13:20.385255485 (14h33m19.974600472s) [upload=15079]: Upload finished. err: <nil> Aug 8 00:14:06.785741109 (14h34m6.375086096s) [upload=15091]: Starting upload Aug 8 00:14:06.785899151 (14h34m6.375244138s) [upload=15091]: Done starting upload Aug 8 00:14:06.786002628 (14h34m6.375347615s) [upload=15091]: Uploading segments... Aug 8 00:14:06.786034078 (14h34m6.375379065s) [upload=15091]: Got next segment. Inline: true Aug 8 00:14:06.786037786 (14h34m6.375382773s) [upload=15091]: Waiting for error group managing segments... Aug 8 00:14:06.786073094 (14h34m6.375418081s) [upload=15091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:14:06.786078945 (14h34m6.375423922s) [upload=15091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:14:06.786081339 (14h34m6.375426326s) [upload=15091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:14:07.092788097 (14h34m6.682133084s) [upload=15091]: Done uploading segments... Aug 8 00:14:07.092803166 (14h34m6.682148153s) [upload=15091]: Upload finished. err: <nil> Duration: 1.852299033s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X...X..........X.......X...X............X....X............] duration: 6.416646508s FAIL (2025-08-08 00:48:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837172821 (14h59m54.426517798s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837385949 (14h59m54.426730936s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837507802 (14h59m54.426852779s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837640145 (14h59m54.426985122s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837819459 (14h59m54.427164446s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837962502 (14h59m54.427307479s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838112790 (14h59m54.427457767s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838238691 (14h59m54.427583678s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838393447 (14h59m54.427738434s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838490402 (14h59m54.427835389s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838660288 (14h59m54.428005275s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838681548 (14h59m54.428026525s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838866312 (14h59m54.428211299s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838893975 (14h59m54.428238962s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.839056986 (14h59m54.428401974s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:55.935384249 (14h59m55.524729246s) [upload=15536]: Next returned nil segment Aug 8 00:39:55.935402815 (14h59m55.524747802s) [upload=15536]: Waiting for error group managing segments... Aug 8 00:39:56.226788010 (14h59m55.816132997s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.265710491 (14h59m55.855055478s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.326363980 (14h59m55.915708967s) [noise=true , address=64.49.13.169:4709 , piece_id=PIPH24KH4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773039652 (14h59m56.362384679s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773213735 (14h59m56.362558732s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.782851711 (14h59m56.372196698s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854376586 (14h59m56.443721573s) [noise=true , address=104.36.82.46:4484 , piece_id=BMHSO2DIXVRRHUZRHWUQ24TXMMTRMVX34SVDUUUMRMKPSUQG3APQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854703231 (14h59m56.444048218s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.914818370 (14h59m56.504163357s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.915843802 (14h59m56.505188849s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.942005621 (14h59m56.531350608s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.981103286 (14h59m56.570448273s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.095564053 (14h59m56.68490904s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216182613 (14h59m56.8055276s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216206869 (14h59m56.805551856s) [seg_pos={0 0}, upload=15536]: Segment reached optimal threshold of 14 pieces. Aug 8 00:39:57.216406632 (14h59m56.805751619s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bN25eEEjxCWKJSr8qj7moDJxArgfsjzPTdBcLdfvoeyxiqPpc): protocol: expected piece hash; context canceled; EOF Aug 8 00:39:57.216460465 (14h59m56.805805452s) [noise=true , address=104.36.82.5:4869 , piece_id=JBCDCG676BBV6U2GAVWDQIWYXKKQGW2DKK6MDY74LNPGP4K3CTQA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection Aug 8 00:39:57.216644467 (14h59m56.805989454s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD): protocol: read tcp 188.166.225.170:50728->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216665567 (14h59m56.806010554s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: read tcp 188.166.225.170:45052->104.36.82.61:4383: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216679534 (14h59m56.806024521s) [seg_pos={0 0}, upload=15536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:39:57.216739699 (14h59m56.806084686s) [upload=15536]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:39:57.216743286 (14h59m56.806088273s) [upload=15536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:39:57.516798834 (14h59m57.106143821s) [upload=15536]: Done uploading segments... Aug 8 00:39:57.516817129 (14h59m57.106162116s) [upload=15536]: Upload finished. err: <nil> Aug 8 00:40:09.579830255 (15h0m9.169175242s) [upload=15541]: Starting upload Aug 8 00:40:09.579877025 (15h0m9.169222012s) [upload=15541]: Done starting upload Aug 8 00:40:09.579941739 (15h0m9.169286726s) [upload=15541]: Uploading segments... Aug 8 00:40:09.580164405 (15h0m9.169509432s) [upload=15541]: Got next segment. Inline: false Aug 8 00:40:09.580219460 (15h0m9.169564477s) [upload=15541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:40:09.580251472 (15h0m9.169596489s) [upload=15541]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:40:09.797368422 (15h0m9.386713409s) [upload=15541]: Next segment err: metaclient: Too Many Requests Aug 8 00:40:09.797398149 (15h0m9.386743136s) [upload=15541]: Done uploading segments... Aug 8 00:40:09.797473614 (15h0m9.386818601s) [upload=15541]: Upload finished. err: metaclient: Too Many Requests Duration: 6.416670634s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.382618521s FAIL (2025-08-08 00:48:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.246137011 (14h21m15.835481998s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.246560380 (14h21m15.835905367s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247408974 (14h21m15.836753961s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247678410 (14h21m15.837023397s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247919882 (14h21m15.837264869s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247943888 (14h21m15.837288875s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248293266 (14h21m15.837638253s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248420540 (14h21m15.837765527s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248557311 (14h21m15.837902288s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248758506 (14h21m15.838103493s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248897792 (14h21m15.838242779s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248989629 (14h21m15.838334616s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249186575 (14h21m15.838531552s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249273772 (14h21m15.838618759s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249459598 (14h21m15.838804585s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249717240 (14h21m15.839062227s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:17.413257645 (14h21m17.002602632s) [upload=14870]: Next returned nil segment Aug 8 00:01:17.413273324 (14h21m17.002618311s) [upload=14870]: Waiting for error group managing segments... Aug 8 00:01:17.710897294 (14h21m17.300242281s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:17.715058896 (14h21m17.304403883s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.126218161 (14h21m17.715563148s) [noise=true , address=79.127.201.215:4356 , piece_id=VWNEIHNI4KPPF4QLW36INIWTUI2CD6MZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.139272367 (14h21m17.728617354s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.221269031 (14h21m17.810614018s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.257350144 (14h21m17.846695121s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.258435089 (14h21m17.847780076s) [noise=true , address=104.36.82.44:4876 , piece_id=BA2G42OXQD3NLONUVD475S2BNC7T37T5PHPE3K3E5HJAJP4W5MKQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.287010923 (14h21m17.87635591s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.348793001 (14h21m17.938137988s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.378129459 (14h21m17.967474436s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.386414901 (14h21m17.975759888s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.516150823 (14h21m18.10549581s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.626647204 (14h21m18.215992191s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:19.042108811 (14h21m18.631453798s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:19.042136654 (14h21m18.631481641s) [seg_pos={0 0}, upload=14870]: Segment reached optimal threshold of 14 pieces. Aug 8 00:01:19.042375541 (14h21m18.631720518s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:19.042485912 (14h21m18.6318309s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: rpc: dial tcp 104.36.82.55:4209: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:01:19.042532361 (14h21m18.631877348s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqNLPRWY7fJRS6F9h4B4qVFv6U5jyWkCy3Um7jfjEdCShEjsD): protocol: write tcp 188.166.225.170:37056->104.36.82.20:4466: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:01:19.042616462 (14h21m18.631961449s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa): protocol: rpc: dial tcp 104.36.82.5:4844: operation was canceled; conn closed: op; piecestore: piecestore close: context canceled Aug 8 00:01:19.042628876 (14h21m18.631973863s) [seg_pos={0 0}, upload=14870]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:01:19.042699029 (14h21m18.632044017s) [upload=14870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:01:19.042703128 (14h21m18.632048115s) [upload=14870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:19.344335316 (14h21m18.933680293s) [upload=14870]: Done uploading segments... Aug 8 00:01:19.344353089 (14h21m18.933698077s) [upload=14870]: Upload finished. err: <nil> Aug 8 00:02:30.778225454 (14h22m30.367570441s) [upload=14892]: Starting upload Aug 8 00:02:30.778283114 (14h22m30.367628151s) [upload=14892]: Done starting upload Aug 8 00:02:30.778411841 (14h22m30.367756828s) [upload=14892]: Uploading segments... Aug 8 00:02:30.778539616 (14h22m30.367884623s) [upload=14892]: Got next segment. Inline: false Aug 8 00:02:30.778583389 (14h22m30.367928376s) [upload=14892]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:30.778587717 (14h22m30.367932704s) [upload=14892]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:03:30.135980490 (14h23m29.725325467s) [upload=14892]: Next segment err: context deadline exceeded Aug 8 00:03:30.136177137 (14h23m29.725522124s) [upload=14892]: Done uploading segments... Aug 8 00:03:30.136196834 (14h23m29.725541821s) [upload=14892]: Upload finished. err: context deadline exceeded Duration: 6.382647426s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X...X..........X.......X...X............X....X...........] duration: 6.10567139s FAIL (2025-08-08 00:48:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837172821 (14h59m54.426517798s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837385949 (14h59m54.426730936s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837507802 (14h59m54.426852779s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837640145 (14h59m54.426985122s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837819459 (14h59m54.427164446s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837962502 (14h59m54.427307479s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838112790 (14h59m54.427457767s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838238691 (14h59m54.427583678s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838393447 (14h59m54.427738434s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838490402 (14h59m54.427835389s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838660288 (14h59m54.428005275s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838681548 (14h59m54.428026525s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838866312 (14h59m54.428211299s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838893975 (14h59m54.428238962s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.839056986 (14h59m54.428401974s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:55.935384249 (14h59m55.524729246s) [upload=15536]: Next returned nil segment Aug 8 00:39:55.935402815 (14h59m55.524747802s) [upload=15536]: Waiting for error group managing segments... Aug 8 00:39:56.226788010 (14h59m55.816132997s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.265710491 (14h59m55.855055478s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.326363980 (14h59m55.915708967s) [noise=true , address=64.49.13.169:4709 , piece_id=PIPH24KH4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773039652 (14h59m56.362384679s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773213735 (14h59m56.362558732s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.782851711 (14h59m56.372196698s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854376586 (14h59m56.443721573s) [noise=true , address=104.36.82.46:4484 , piece_id=BMHSO2DIXVRRHUZRHWUQ24TXMMTRMVX34SVDUUUMRMKPSUQG3APQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854703231 (14h59m56.444048218s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.914818370 (14h59m56.504163357s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.915843802 (14h59m56.505188849s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.942005621 (14h59m56.531350608s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.981103286 (14h59m56.570448273s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.095564053 (14h59m56.68490904s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216182613 (14h59m56.8055276s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216206869 (14h59m56.805551856s) [seg_pos={0 0}, upload=15536]: Segment reached optimal threshold of 14 pieces. Aug 8 00:39:57.216406632 (14h59m56.805751619s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bN25eEEjxCWKJSr8qj7moDJxArgfsjzPTdBcLdfvoeyxiqPpc): protocol: expected piece hash; context canceled; EOF Aug 8 00:39:57.216460465 (14h59m56.805805452s) [noise=true , address=104.36.82.5:4869 , piece_id=JBCDCG676BBV6U2GAVWDQIWYXKKQGW2DKK6MDY74LNPGP4K3CTQA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection Aug 8 00:39:57.216644467 (14h59m56.805989454s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD): protocol: read tcp 188.166.225.170:50728->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216665567 (14h59m56.806010554s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: read tcp 188.166.225.170:45052->104.36.82.61:4383: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216679534 (14h59m56.806024521s) [seg_pos={0 0}, upload=15536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:39:57.216739699 (14h59m56.806084686s) [upload=15536]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:39:57.216743286 (14h59m56.806088273s) [upload=15536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:39:57.516798834 (14h59m57.106143821s) [upload=15536]: Done uploading segments... Aug 8 00:39:57.516817129 (14h59m57.106162116s) [upload=15536]: Upload finished. err: <nil> Aug 8 00:40:09.579830255 (15h0m9.169175242s) [upload=15541]: Starting upload Aug 8 00:40:09.579877025 (15h0m9.169222012s) [upload=15541]: Done starting upload Aug 8 00:40:09.579941739 (15h0m9.169286726s) [upload=15541]: Uploading segments... Aug 8 00:40:09.580164405 (15h0m9.169509432s) [upload=15541]: Got next segment. Inline: false Aug 8 00:40:09.580219460 (15h0m9.169564477s) [upload=15541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:40:09.580251472 (15h0m9.169596489s) [upload=15541]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:40:09.797368422 (15h0m9.386713409s) [upload=15541]: Next segment err: metaclient: Too Many Requests Aug 8 00:40:09.797398149 (15h0m9.386743136s) [upload=15541]: Done uploading segments... Aug 8 00:40:09.797473614 (15h0m9.386818601s) [upload=15541]: Upload finished. err: metaclient: Too Many Requests Duration: 6.105710985s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.871237827s FAIL (2025-08-08 00:48:04 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== =14964]: Done uploading segments... Aug 8 00:06:41.669168196 (14h26m41.258513183s) [upload=14964]: Upload finished. err: <nil> Aug 8 00:07:06.258431851 (14h27m5.847776838s) [upload=14976]: Starting upload Aug 8 00:07:06.258508298 (14h27m5.847853285s) [upload=14976]: Done starting upload Aug 8 00:07:06.258574986 (14h27m5.847919983s) [upload=14976]: Uploading segments... Aug 8 00:07:06.258618599 (14h27m5.847963616s) [upload=14976]: Got next segment. Inline: true Aug 8 00:07:06.258624600 (14h27m5.847969587s) [upload=14976]: Waiting for error group managing segments... Aug 8 00:07:06.258705555 (14h27m5.848050542s) [upload=14976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:07:06.258712268 (14h27m5.848057296s) [upload=14976]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:07:06.258716226 (14h27m5.848061253s) [upload=14976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:07:06.522670976 (14h27m6.112015963s) [upload=14976]: Done uploading segments... Aug 8 00:07:06.522717766 (14h27m6.112062773s) [upload=14976]: Upload finished. err: <nil> Aug 8 00:07:36.371369594 (14h27m35.960714581s) [upload=14980]: Starting upload Aug 8 00:07:36.371417074 (14h27m35.960762061s) [upload=14980]: Done starting upload Aug 8 00:07:36.371466890 (14h27m35.960811877s) [upload=14980]: Uploading segments... Aug 8 00:07:36.371508951 (14h27m35.960853938s) [upload=14980]: Got next segment. Inline: true Aug 8 00:07:36.371512888 (14h27m35.960857875s) [upload=14980]: Waiting for error group managing segments... Aug 8 00:07:36.371543446 (14h27m35.960888433s) [upload=14980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:07:36.371547434 (14h27m35.960892421s) [upload=14980]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:07:36.371549708 (14h27m35.960894695s) [upload=14980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:07:36.789946570 (14h27m36.379291557s) [upload=14980]: Done uploading segments... Aug 8 00:07:36.789964876 (14h27m36.379309863s) [upload=14980]: Upload finished. err: <nil> Aug 8 00:08:16.866388064 (14h28m16.455733051s) [upload=14994]: Starting upload Aug 8 00:08:16.866430886 (14h28m16.455775873s) [upload=14994]: Done starting upload Aug 8 00:08:16.866496882 (14h28m16.455841869s) [upload=14994]: Uploading segments... Aug 8 00:08:16.866525727 (14h28m16.455870714s) [upload=14994]: Got next segment. Inline: true Aug 8 00:08:16.866529494 (14h28m16.455874481s) [upload=14994]: Waiting for error group managing segments... Aug 8 00:08:16.866570884 (14h28m16.455915871s) [upload=14994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:08:16.866575212 (14h28m16.455920199s) [upload=14994]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:08:16.866577626 (14h28m16.455922613s) [upload=14994]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:08:17.138322091 (14h28m16.727667078s) [upload=14994]: Done uploading segments... Aug 8 00:08:17.138338402 (14h28m16.727683389s) [upload=14994]: Upload finished. err: <nil> Aug 8 00:08:57.633625328 (14h28m57.222970315s) [upload=15006]: Starting upload Aug 8 00:08:57.633689390 (14h28m57.223034378s) [upload=15006]: Done starting upload Aug 8 00:08:57.633744837 (14h28m57.223089824s) [upload=15006]: Uploading segments... Aug 8 00:08:57.633775525 (14h28m57.223120512s) [upload=15006]: Got next segment. Inline: true Aug 8 00:08:57.633779222 (14h28m57.223124199s) [upload=15006]: Waiting for error group managing segments... Aug 8 00:08:57.633951743 (14h28m57.22329672s) [upload=15006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:08:57.633956161 (14h28m57.223301148s) [upload=15006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:08:57.633958886 (14h28m57.223303873s) [upload=15006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:08:57.920842105 (14h28m57.510187102s) [upload=15006]: Done uploading segments... Aug 8 00:08:57.920865330 (14h28m57.510210327s) [upload=15006]: Upload finished. err: <nil> Aug 8 00:09:43.341371442 (14h29m42.930716449s) [upload=15015]: Starting upload Aug 8 00:09:43.341457767 (14h29m42.930802774s) [upload=15015]: Done starting upload Aug 8 00:09:43.341511951 (14h29m42.930856938s) [upload=15015]: Uploading segments... Aug 8 00:09:43.341579249 (14h29m42.930924236s) [upload=15015]: Got next segment. Inline: true Aug 8 00:09:43.341585642 (14h29m42.930930639s) [upload=15015]: Waiting for error group managing segments... Aug 8 00:09:43.341640437 (14h29m42.930985454s) [upload=15015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:09:43.341648422 (14h29m42.930993469s) [upload=15015]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:09:43.341653391 (14h29m42.930998408s) [upload=15015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:09:43.627931533 (14h29m43.21727652s) [upload=15015]: Done uploading segments... Aug 8 00:09:43.627948425 (14h29m43.217293402s) [upload=15015]: Upload finished. err: <nil> Aug 8 00:10:45.166177580 (14h30m44.755522568s) [upload=15031]: Starting upload Aug 8 00:10:45.166220523 (14h30m44.75556551s) [upload=15031]: Done starting upload Aug 8 00:10:45.166273034 (14h30m44.755618011s) [upload=15031]: Uploading segments... Aug 8 00:10:45.166297991 (14h30m44.755642979s) [upload=15031]: Got next segment. Inline: true Aug 8 00:10:45.166301568 (14h30m44.755646555s) [upload=15031]: Waiting for error group managing segments... Aug 8 00:10:45.166332277 (14h30m44.755677264s) [upload=15031]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:10:45.166335783 (14h30m44.75568077s) [upload=15031]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:10:45.166337998 (14h30m44.755682985s) [upload=15031]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:10:45.444515471 (14h30m45.033860458s) [upload=15031]: Done uploading segments... Aug 8 00:10:45.444533305 (14h30m45.033878282s) [upload=15031]: Upload finished. err: <nil> Aug 8 00:11:54.979425886 (14h31m54.568770863s) [upload=15052]: Starting upload Aug 8 00:11:54.979471082 (14h31m54.568816069s) [upload=15052]: Done starting upload Aug 8 00:11:54.979517260 (14h31m54.568862237s) [upload=15052]: Uploading segments... Aug 8 00:11:54.979549422 (14h31m54.568894399s) [upload=15052]: Got next segment. Inline: true Aug 8 00:11:54.979553008 (14h31m54.568897995s) [upload=15052]: Waiting for error group managing segments... Aug 8 00:11:54.979584328 (14h31m54.568929315s) [upload=15052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:11:54.979588356 (14h31m54.568933333s) [upload=15052]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:11:54.979590661 (14h31m54.568935648s) [upload=15052]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:11:55.340902393 (14h31m54.93024738s) [upload=15052]: Done uploading segments... Aug 8 00:11:55.340920908 (14h31m54.930265895s) [upload=15052]: Upload finished. err: <nil> Aug 8 00:12:44.503574128 (14h32m44.092919115s) [upload=15065]: Starting upload Aug 8 00:12:44.503634083 (14h32m44.09297907s) [upload=15065]: Done starting upload Aug 8 00:12:44.503710068 (14h32m44.093055055s) [upload=15065]: Uploading segments... Aug 8 00:12:44.503762669 (14h32m44.093107656s) [upload=15065]: Got next segment. Inline: true Aug 8 00:12:44.503767698 (14h32m44.093112685s) [upload=15065]: Waiting for error group managing segments... Aug 8 00:12:44.503805531 (14h32m44.093150508s) [upload=15065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:12:44.503809379 (14h32m44.093154366s) [upload=15065]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:12:44.503811633 (14h32m44.09315662s) [upload=15065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:12:44.792734473 (14h32m44.38207946s) [upload=15065]: Done uploading segments... Aug 8 00:12:44.792752288 (14h32m44.382097275s) [upload=15065]: Upload finished. err: <nil> Aug 8 00:13:19.961146705 (14h33m19.550491693s) [upload=15079]: Starting upload Aug 8 00:13:19.961187835 (14h33m19.550532822s) [upload=15079]: Done starting upload Aug 8 00:13:19.961236618 (14h33m19.550581605s) [upload=15079]: Uploading segments... Aug 8 00:13:19.961268168 (14h33m19.550613145s) [upload=15079]: Got next segment. Inline: true Aug 8 00:13:19.961273849 (14h33m19.550618836s) [upload=15079]: Waiting for error group managing segments... Aug 8 00:13:19.961304548 (14h33m19.550649525s) [upload=15079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:13:19.961308345 (14h33m19.550653332s) [upload=15079]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:13:19.961310910 (14h33m19.550655887s) [upload=15079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:13:20.385237160 (14h33m19.974582137s) [upload=15079]: Done uploading segments... Aug 8 00:13:20.385255485 (14h33m19.974600472s) [upload=15079]: Upload finished. err: <nil> Aug 8 00:14:06.785741109 (14h34m6.375086096s) [upload=15091]: Starting upload Aug 8 00:14:06.785899151 (14h34m6.375244138s) [upload=15091]: Done starting upload Aug 8 00:14:06.786002628 (14h34m6.375347615s) [upload=15091]: Uploading segments... Aug 8 00:14:06.786034078 (14h34m6.375379065s) [upload=15091]: Got next segment. Inline: true Aug 8 00:14:06.786037786 (14h34m6.375382773s) [upload=15091]: Waiting for error group managing segments... Aug 8 00:14:06.786073094 (14h34m6.375418081s) [upload=15091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:14:06.786078945 (14h34m6.375423922s) [upload=15091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:14:06.786081339 (14h34m6.375426326s) [upload=15091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:14:07.092788097 (14h34m6.682133084s) [upload=15091]: Done uploading segments... Aug 8 00:14:07.092803166 (14h34m6.682148153s) [upload=15091]: Upload finished. err: <nil> Duration: 1.87127121s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [XXXX....X...................................................] duration: 1.884499593s FAIL (2025-08-08 00:47:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9109253s) [upload=14753]: Starting upload Aug 7 23:54:04.489828037 (14h14m4.079173014s) [upload=14753]: Done starting upload Aug 7 23:54:04.489877612 (14h14m4.079222589s) [upload=14753]: Uploading segments... Aug 7 23:54:04.489903933 (14h14m4.07924892s) [upload=14753]: Got next segment. Inline: true Aug 7 23:54:04.489907469 (14h14m4.079252446s) [upload=14753]: Waiting for error group managing segments... Aug 7 23:54:04.489939651 (14h14m4.079284638s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:04.489943298 (14h14m4.079288285s) [upload=14753]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:04.489945472 (14h14m4.079290459s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:04.778726062 (14h14m4.368071049s) [upload=14753]: Done uploading segments... Aug 7 23:54:04.778744989 (14h14m4.368089976s) [upload=14753]: Upload finished. err: <nil> Aug 7 23:55:14.199719855 (14h15m13.789064892s) [upload=14770]: Starting upload Aug 7 23:55:14.199777645 (14h15m13.789122632s) [upload=14770]: Done starting upload Aug 7 23:55:14.199824415 (14h15m13.789169392s) [upload=14770]: Uploading segments... Aug 7 23:55:14.199852819 (14h15m13.789197806s) [upload=14770]: Got next segment. Inline: true Aug 7 23:55:14.199856436 (14h15m13.789201423s) [upload=14770]: Waiting for error group managing segments... Aug 7 23:55:14.199889419 (14h15m13.789234406s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:14.199893557 (14h15m13.789238544s) [upload=14770]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:14.199895951 (14h15m13.789240938s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:13.542516296 (14h16m13.131861283s) [upload=14770]: Done uploading segments... Aug 7 23:56:13.542536334 (14h16m13.131881321s) [upload=14770]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:14.428677736 (14h16m14.018022723s) [upload=14788]: Starting upload Aug 7 23:56:14.428730978 (14h16m14.018075965s) [upload=14788]: Done starting upload Aug 7 23:56:14.428789289 (14h16m14.018134276s) [upload=14788]: Uploading segments... Aug 7 23:56:14.428819026 (14h16m14.018164013s) [upload=14788]: Got next segment. Inline: true Aug 7 23:56:14.428822873 (14h16m14.01816785s) [upload=14788]: Waiting for error group managing segments... Aug 7 23:56:14.428855215 (14h16m14.018200202s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:14.428858562 (14h16m14.018203549s) [upload=14788]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:14.428860736 (14h16m14.018205723s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:13.781984834 (14h17m13.371329821s) [upload=14788]: Done uploading segments... Aug 7 23:57:13.782003370 (14h17m13.371348357s) [upload=14788]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:29.079359478 (14h17m28.668704466s) [upload=14808]: Starting upload Aug 7 23:57:29.079416147 (14h17m28.668761134s) [upload=14808]: Done starting upload Aug 7 23:57:29.079461845 (14h17m28.668806822s) [upload=14808]: Uploading segments... Aug 7 23:57:29.079488876 (14h17m28.668833863s) [upload=14808]: Got next segment. Inline: true Aug 7 23:57:29.079492263 (14h17m28.66883725s) [upload=14808]: Waiting for error group managing segments... Aug 7 23:57:29.079523392 (14h17m28.668868379s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:29.079527049 (14h17m28.668872036s) [upload=14808]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:29.079529484 (14h17m28.668874461s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:28.439377526 (14h18m28.028722513s) [upload=14808]: Done uploading segments... Aug 7 23:58:28.439395721 (14h18m28.028740708s) [upload=14808]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:29.333669502 (14h18m28.923014489s) [upload=14825]: Starting upload Aug 7 23:58:29.333742581 (14h18m28.923087568s) [upload=14825]: Done starting upload Aug 7 23:58:29.333835740 (14h18m28.923180727s) [upload=14825]: Uploading segments... Aug 7 23:58:29.333878000 (14h18m28.923222987s) [upload=14825]: Got next segment. Inline: true Aug 7 23:58:29.333881998 (14h18m28.923226975s) [upload=14825]: Waiting for error group managing segments... Aug 7 23:58:29.333911455 (14h18m28.923256442s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:29.333917496 (14h18m28.923262473s) [upload=14825]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:29.333919741 (14h18m28.923264728s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:28.692415040 (14h19m28.281760027s) [upload=14825]: Done uploading segments... Aug 7 23:59:28.692444777 (14h19m28.281789764s) [upload=14825]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:29.562470053 (14h19m29.15181504s) [upload=14840]: Starting upload Aug 7 23:59:29.562538985 (14h19m29.151883972s) [upload=14840]: Done starting upload Aug 7 23:59:29.562614629 (14h19m29.151959616s) [upload=14840]: Uploading segments... Aug 7 23:59:29.562650589 (14h19m29.151995566s) [upload=14840]: Got next segment. Inline: true Aug 7 23:59:29.562655498 (14h19m29.152000485s) [upload=14840]: Waiting for error group managing segments... Aug 7 23:59:29.562686507 (14h19m29.152031494s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:29.562690264 (14h19m29.152035241s) [upload=14840]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:29.562692428 (14h19m29.152037405s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:29.822497550 (14h19m29.411842537s) [upload=14840]: Done uploading segments... Aug 7 23:59:29.822514693 (14h19m29.41185968s) [upload=14840]: Upload finished. err: <nil> Aug 8 00:00:35.820554363 (14h20m35.40989935s) [upload=14856]: Starting upload Aug 8 00:00:35.820610611 (14h20m35.409955598s) [upload=14856]: Done starting upload Aug 8 00:00:35.820669503 (14h20m35.4100145s) [upload=14856]: Uploading segments... Aug 8 00:00:35.820742333 (14h20m35.41008732s) [upload=14856]: Got next segment. Inline: true Aug 8 00:00:35.820747282 (14h20m35.410092269s) [upload=14856]: Waiting for error group managing segments... Aug 8 00:00:35.820788371 (14h20m35.410133358s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.820793050 (14h20m35.410138037s) [upload=14856]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.820795485 (14h20m35.410140472s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:36.084321816 (14h20m35.673666803s) [upload=14856]: Done uploading segments... Aug 8 00:00:36.084338618 (14h20m35.673683615s) [upload=14856]: Upload finished. err: <nil> Aug 8 00:01:14.496466774 (14h21m14.085811781s) [upload=14868]: Starting upload Aug 8 00:01:14.496556174 (14h21m14.085901162s) [upload=14868]: Done starting upload Aug 8 00:01:14.496609206 (14h21m14.085954183s) [upload=14868]: Uploading segments... Aug 8 00:01:14.496639985 (14h21m14.085984972s) [upload=14868]: Got next segment. Inline: true Aug 8 00:01:14.496643592 (14h21m14.085988579s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:01:14.496678638 (14h21m14.086023625s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:14.496682386 (14h21m14.086027373s) [upload=14868]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:14.496684681 (14h21m14.086029658s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:14.760681029 (14h21m14.350026016s) [upload=14868]: Done uploading segments... Aug 8 00:01:14.760702580 (14h21m14.350047567s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:01:50.577049881 (14h21m50.166394858s) [upload=14878]: Starting upload Aug 8 00:01:50.577100538 (14h21m50.166445525s) [upload=14878]: Done starting upload Aug 8 00:01:50.577180091 (14h21m50.166525078s) [upload=14878]: Uploading segments... Aug 8 00:01:50.577228012 (14h21m50.166573s) [upload=14878]: Got next segment. Inline: true Aug 8 00:01:50.577231469 (14h21m50.166576456s) [upload=14878]: Waiting for error group managing segments... Aug 8 00:01:50.577270633 (14h21m50.166615621s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:50.577274721 (14h21m50.166619708s) [upload=14878]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:50.577276906 (14h21m50.166621893s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:50.863237339 (14h21m50.452582326s) [upload=14878]: Done uploading segments... Aug 8 00:01:50.863254973 (14h21m50.452599961s) [upload=14878]: Upload finished. err: <nil> Aug 8 00:02:41.474648957 (14h22m41.063993944s) [upload=14899]: Starting upload Aug 8 00:02:41.474716297 (14h22m41.064061274s) [upload=14899]: Done starting upload Aug 8 00:02:41.474771743 (14h22m41.06411673s) [upload=14899]: Uploading segments... Aug 8 00:02:41.474801099 (14h22m41.064146086s) [upload=14899]: Got next segment. Inline: true Aug 8 00:02:41.474804966 (14h22m41.064149953s) [upload=14899]: Waiting for error group managing segments... Aug 8 00:02:41.474841336 (14h22m41.064186323s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:41.474845214 (14h22m41.064190201s) [upload=14899]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:41.474847558 (14h22m41.064192535s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:03:40.835731182 (14h23m40.425076189s) [upload=14899]: Done uploading segments... Aug 8 00:03:40.835778212 (14h23m40.425123199s) [upload=14899]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.88455074s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X...X..........X.......X...X............X....X..........] duration: 6.361328788s FAIL (2025-08-08 00:47:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837172821 (14h59m54.426517798s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837385949 (14h59m54.426730936s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837507802 (14h59m54.426852779s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837640145 (14h59m54.426985122s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837819459 (14h59m54.427164446s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837962502 (14h59m54.427307479s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838112790 (14h59m54.427457767s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838238691 (14h59m54.427583678s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838393447 (14h59m54.427738434s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838490402 (14h59m54.427835389s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838660288 (14h59m54.428005275s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838681548 (14h59m54.428026525s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838866312 (14h59m54.428211299s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838893975 (14h59m54.428238962s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.839056986 (14h59m54.428401974s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:55.935384249 (14h59m55.524729246s) [upload=15536]: Next returned nil segment Aug 8 00:39:55.935402815 (14h59m55.524747802s) [upload=15536]: Waiting for error group managing segments... Aug 8 00:39:56.226788010 (14h59m55.816132997s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.265710491 (14h59m55.855055478s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.326363980 (14h59m55.915708967s) [noise=true , address=64.49.13.169:4709 , piece_id=PIPH24KH4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773039652 (14h59m56.362384679s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773213735 (14h59m56.362558732s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.782851711 (14h59m56.372196698s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854376586 (14h59m56.443721573s) [noise=true , address=104.36.82.46:4484 , piece_id=BMHSO2DIXVRRHUZRHWUQ24TXMMTRMVX34SVDUUUMRMKPSUQG3APQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854703231 (14h59m56.444048218s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.914818370 (14h59m56.504163357s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.915843802 (14h59m56.505188849s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.942005621 (14h59m56.531350608s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.981103286 (14h59m56.570448273s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.095564053 (14h59m56.68490904s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216182613 (14h59m56.8055276s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216206869 (14h59m56.805551856s) [seg_pos={0 0}, upload=15536]: Segment reached optimal threshold of 14 pieces. Aug 8 00:39:57.216406632 (14h59m56.805751619s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bN25eEEjxCWKJSr8qj7moDJxArgfsjzPTdBcLdfvoeyxiqPpc): protocol: expected piece hash; context canceled; EOF Aug 8 00:39:57.216460465 (14h59m56.805805452s) [noise=true , address=104.36.82.5:4869 , piece_id=JBCDCG676BBV6U2GAVWDQIWYXKKQGW2DKK6MDY74LNPGP4K3CTQA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection Aug 8 00:39:57.216644467 (14h59m56.805989454s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD): protocol: read tcp 188.166.225.170:50728->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216665567 (14h59m56.806010554s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: read tcp 188.166.225.170:45052->104.36.82.61:4383: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216679534 (14h59m56.806024521s) [seg_pos={0 0}, upload=15536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:39:57.216739699 (14h59m56.806084686s) [upload=15536]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:39:57.216743286 (14h59m56.806088273s) [upload=15536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:39:57.516798834 (14h59m57.106143821s) [upload=15536]: Done uploading segments... Aug 8 00:39:57.516817129 (14h59m57.106162116s) [upload=15536]: Upload finished. err: <nil> Aug 8 00:40:09.579830255 (15h0m9.169175242s) [upload=15541]: Starting upload Aug 8 00:40:09.579877025 (15h0m9.169222012s) [upload=15541]: Done starting upload Aug 8 00:40:09.579941739 (15h0m9.169286726s) [upload=15541]: Uploading segments... Aug 8 00:40:09.580164405 (15h0m9.169509432s) [upload=15541]: Got next segment. Inline: false Aug 8 00:40:09.580219460 (15h0m9.169564477s) [upload=15541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:40:09.580251472 (15h0m9.169596489s) [upload=15541]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:40:09.797368422 (15h0m9.386713409s) [upload=15541]: Next segment err: metaclient: Too Many Requests Aug 8 00:40:09.797398149 (15h0m9.386743136s) [upload=15541]: Done uploading segments... Aug 8 00:40:09.797473614 (15h0m9.386818601s) [upload=15541]: Upload finished. err: metaclient: Too Many Requests Duration: 6.36135647s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.104343798s FAIL (2025-08-08 00:47:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.246137011 (14h21m15.835481998s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.246560380 (14h21m15.835905367s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247408974 (14h21m15.836753961s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247678410 (14h21m15.837023397s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247919882 (14h21m15.837264869s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247943888 (14h21m15.837288875s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248293266 (14h21m15.837638253s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248420540 (14h21m15.837765527s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248557311 (14h21m15.837902288s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248758506 (14h21m15.838103493s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248897792 (14h21m15.838242779s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248989629 (14h21m15.838334616s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249186575 (14h21m15.838531552s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249273772 (14h21m15.838618759s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249459598 (14h21m15.838804585s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249717240 (14h21m15.839062227s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:17.413257645 (14h21m17.002602632s) [upload=14870]: Next returned nil segment Aug 8 00:01:17.413273324 (14h21m17.002618311s) [upload=14870]: Waiting for error group managing segments... Aug 8 00:01:17.710897294 (14h21m17.300242281s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:17.715058896 (14h21m17.304403883s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.126218161 (14h21m17.715563148s) [noise=true , address=79.127.201.215:4356 , piece_id=VWNEIHNI4KPPF4QLW36INIWTUI2CD6MZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.139272367 (14h21m17.728617354s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.221269031 (14h21m17.810614018s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.257350144 (14h21m17.846695121s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.258435089 (14h21m17.847780076s) [noise=true , address=104.36.82.44:4876 , piece_id=BA2G42OXQD3NLONUVD475S2BNC7T37T5PHPE3K3E5HJAJP4W5MKQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.287010923 (14h21m17.87635591s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.348793001 (14h21m17.938137988s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.378129459 (14h21m17.967474436s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.386414901 (14h21m17.975759888s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.516150823 (14h21m18.10549581s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.626647204 (14h21m18.215992191s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:19.042108811 (14h21m18.631453798s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:19.042136654 (14h21m18.631481641s) [seg_pos={0 0}, upload=14870]: Segment reached optimal threshold of 14 pieces. Aug 8 00:01:19.042375541 (14h21m18.631720518s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:19.042485912 (14h21m18.6318309s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: rpc: dial tcp 104.36.82.55:4209: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:01:19.042532361 (14h21m18.631877348s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqNLPRWY7fJRS6F9h4B4qVFv6U5jyWkCy3Um7jfjEdCShEjsD): protocol: write tcp 188.166.225.170:37056->104.36.82.20:4466: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:01:19.042616462 (14h21m18.631961449s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa): protocol: rpc: dial tcp 104.36.82.5:4844: operation was canceled; conn closed: op; piecestore: piecestore close: context canceled Aug 8 00:01:19.042628876 (14h21m18.631973863s) [seg_pos={0 0}, upload=14870]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:01:19.042699029 (14h21m18.632044017s) [upload=14870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:01:19.042703128 (14h21m18.632048115s) [upload=14870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:19.344335316 (14h21m18.933680293s) [upload=14870]: Done uploading segments... Aug 8 00:01:19.344353089 (14h21m18.933698077s) [upload=14870]: Upload finished. err: <nil> Aug 8 00:02:30.778225454 (14h22m30.367570441s) [upload=14892]: Starting upload Aug 8 00:02:30.778283114 (14h22m30.367628151s) [upload=14892]: Done starting upload Aug 8 00:02:30.778411841 (14h22m30.367756828s) [upload=14892]: Uploading segments... Aug 8 00:02:30.778539616 (14h22m30.367884623s) [upload=14892]: Got next segment. Inline: false Aug 8 00:02:30.778583389 (14h22m30.367928376s) [upload=14892]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:30.778587717 (14h22m30.367932704s) [upload=14892]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:03:30.135980490 (14h23m29.725325467s) [upload=14892]: Next segment err: context deadline exceeded Aug 8 00:03:30.136177137 (14h23m29.725522124s) [upload=14892]: Done uploading segments... Aug 8 00:03:30.136196834 (14h23m29.725541821s) [upload=14892]: Upload finished. err: context deadline exceeded Duration: 6.104372783s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.820857667s FAIL (2025-08-08 00:46:57 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== =14964]: Done uploading segments... Aug 8 00:06:41.669168196 (14h26m41.258513183s) [upload=14964]: Upload finished. err: <nil> Aug 8 00:07:06.258431851 (14h27m5.847776838s) [upload=14976]: Starting upload Aug 8 00:07:06.258508298 (14h27m5.847853285s) [upload=14976]: Done starting upload Aug 8 00:07:06.258574986 (14h27m5.847919983s) [upload=14976]: Uploading segments... Aug 8 00:07:06.258618599 (14h27m5.847963616s) [upload=14976]: Got next segment. Inline: true Aug 8 00:07:06.258624600 (14h27m5.847969587s) [upload=14976]: Waiting for error group managing segments... Aug 8 00:07:06.258705555 (14h27m5.848050542s) [upload=14976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:07:06.258712268 (14h27m5.848057296s) [upload=14976]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:07:06.258716226 (14h27m5.848061253s) [upload=14976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:07:06.522670976 (14h27m6.112015963s) [upload=14976]: Done uploading segments... Aug 8 00:07:06.522717766 (14h27m6.112062773s) [upload=14976]: Upload finished. err: <nil> Aug 8 00:07:36.371369594 (14h27m35.960714581s) [upload=14980]: Starting upload Aug 8 00:07:36.371417074 (14h27m35.960762061s) [upload=14980]: Done starting upload Aug 8 00:07:36.371466890 (14h27m35.960811877s) [upload=14980]: Uploading segments... Aug 8 00:07:36.371508951 (14h27m35.960853938s) [upload=14980]: Got next segment. Inline: true Aug 8 00:07:36.371512888 (14h27m35.960857875s) [upload=14980]: Waiting for error group managing segments... Aug 8 00:07:36.371543446 (14h27m35.960888433s) [upload=14980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:07:36.371547434 (14h27m35.960892421s) [upload=14980]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:07:36.371549708 (14h27m35.960894695s) [upload=14980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:07:36.789946570 (14h27m36.379291557s) [upload=14980]: Done uploading segments... Aug 8 00:07:36.789964876 (14h27m36.379309863s) [upload=14980]: Upload finished. err: <nil> Aug 8 00:08:16.866388064 (14h28m16.455733051s) [upload=14994]: Starting upload Aug 8 00:08:16.866430886 (14h28m16.455775873s) [upload=14994]: Done starting upload Aug 8 00:08:16.866496882 (14h28m16.455841869s) [upload=14994]: Uploading segments... Aug 8 00:08:16.866525727 (14h28m16.455870714s) [upload=14994]: Got next segment. Inline: true Aug 8 00:08:16.866529494 (14h28m16.455874481s) [upload=14994]: Waiting for error group managing segments... Aug 8 00:08:16.866570884 (14h28m16.455915871s) [upload=14994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:08:16.866575212 (14h28m16.455920199s) [upload=14994]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:08:16.866577626 (14h28m16.455922613s) [upload=14994]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:08:17.138322091 (14h28m16.727667078s) [upload=14994]: Done uploading segments... Aug 8 00:08:17.138338402 (14h28m16.727683389s) [upload=14994]: Upload finished. err: <nil> Aug 8 00:08:57.633625328 (14h28m57.222970315s) [upload=15006]: Starting upload Aug 8 00:08:57.633689390 (14h28m57.223034378s) [upload=15006]: Done starting upload Aug 8 00:08:57.633744837 (14h28m57.223089824s) [upload=15006]: Uploading segments... Aug 8 00:08:57.633775525 (14h28m57.223120512s) [upload=15006]: Got next segment. Inline: true Aug 8 00:08:57.633779222 (14h28m57.223124199s) [upload=15006]: Waiting for error group managing segments... Aug 8 00:08:57.633951743 (14h28m57.22329672s) [upload=15006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:08:57.633956161 (14h28m57.223301148s) [upload=15006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:08:57.633958886 (14h28m57.223303873s) [upload=15006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:08:57.920842105 (14h28m57.510187102s) [upload=15006]: Done uploading segments... Aug 8 00:08:57.920865330 (14h28m57.510210327s) [upload=15006]: Upload finished. err: <nil> Aug 8 00:09:43.341371442 (14h29m42.930716449s) [upload=15015]: Starting upload Aug 8 00:09:43.341457767 (14h29m42.930802774s) [upload=15015]: Done starting upload Aug 8 00:09:43.341511951 (14h29m42.930856938s) [upload=15015]: Uploading segments... Aug 8 00:09:43.341579249 (14h29m42.930924236s) [upload=15015]: Got next segment. Inline: true Aug 8 00:09:43.341585642 (14h29m42.930930639s) [upload=15015]: Waiting for error group managing segments... Aug 8 00:09:43.341640437 (14h29m42.930985454s) [upload=15015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:09:43.341648422 (14h29m42.930993469s) [upload=15015]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:09:43.341653391 (14h29m42.930998408s) [upload=15015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:09:43.627931533 (14h29m43.21727652s) [upload=15015]: Done uploading segments... Aug 8 00:09:43.627948425 (14h29m43.217293402s) [upload=15015]: Upload finished. err: <nil> Aug 8 00:10:45.166177580 (14h30m44.755522568s) [upload=15031]: Starting upload Aug 8 00:10:45.166220523 (14h30m44.75556551s) [upload=15031]: Done starting upload Aug 8 00:10:45.166273034 (14h30m44.755618011s) [upload=15031]: Uploading segments... Aug 8 00:10:45.166297991 (14h30m44.755642979s) [upload=15031]: Got next segment. Inline: true Aug 8 00:10:45.166301568 (14h30m44.755646555s) [upload=15031]: Waiting for error group managing segments... Aug 8 00:10:45.166332277 (14h30m44.755677264s) [upload=15031]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:10:45.166335783 (14h30m44.75568077s) [upload=15031]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:10:45.166337998 (14h30m44.755682985s) [upload=15031]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:10:45.444515471 (14h30m45.033860458s) [upload=15031]: Done uploading segments... Aug 8 00:10:45.444533305 (14h30m45.033878282s) [upload=15031]: Upload finished. err: <nil> Aug 8 00:11:54.979425886 (14h31m54.568770863s) [upload=15052]: Starting upload Aug 8 00:11:54.979471082 (14h31m54.568816069s) [upload=15052]: Done starting upload Aug 8 00:11:54.979517260 (14h31m54.568862237s) [upload=15052]: Uploading segments... Aug 8 00:11:54.979549422 (14h31m54.568894399s) [upload=15052]: Got next segment. Inline: true Aug 8 00:11:54.979553008 (14h31m54.568897995s) [upload=15052]: Waiting for error group managing segments... Aug 8 00:11:54.979584328 (14h31m54.568929315s) [upload=15052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:11:54.979588356 (14h31m54.568933333s) [upload=15052]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:11:54.979590661 (14h31m54.568935648s) [upload=15052]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:11:55.340902393 (14h31m54.93024738s) [upload=15052]: Done uploading segments... Aug 8 00:11:55.340920908 (14h31m54.930265895s) [upload=15052]: Upload finished. err: <nil> Aug 8 00:12:44.503574128 (14h32m44.092919115s) [upload=15065]: Starting upload Aug 8 00:12:44.503634083 (14h32m44.09297907s) [upload=15065]: Done starting upload Aug 8 00:12:44.503710068 (14h32m44.093055055s) [upload=15065]: Uploading segments... Aug 8 00:12:44.503762669 (14h32m44.093107656s) [upload=15065]: Got next segment. Inline: true Aug 8 00:12:44.503767698 (14h32m44.093112685s) [upload=15065]: Waiting for error group managing segments... Aug 8 00:12:44.503805531 (14h32m44.093150508s) [upload=15065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:12:44.503809379 (14h32m44.093154366s) [upload=15065]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:12:44.503811633 (14h32m44.09315662s) [upload=15065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:12:44.792734473 (14h32m44.38207946s) [upload=15065]: Done uploading segments... Aug 8 00:12:44.792752288 (14h32m44.382097275s) [upload=15065]: Upload finished. err: <nil> Aug 8 00:13:19.961146705 (14h33m19.550491693s) [upload=15079]: Starting upload Aug 8 00:13:19.961187835 (14h33m19.550532822s) [upload=15079]: Done starting upload Aug 8 00:13:19.961236618 (14h33m19.550581605s) [upload=15079]: Uploading segments... Aug 8 00:13:19.961268168 (14h33m19.550613145s) [upload=15079]: Got next segment. Inline: true Aug 8 00:13:19.961273849 (14h33m19.550618836s) [upload=15079]: Waiting for error group managing segments... Aug 8 00:13:19.961304548 (14h33m19.550649525s) [upload=15079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:13:19.961308345 (14h33m19.550653332s) [upload=15079]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:13:19.961310910 (14h33m19.550655887s) [upload=15079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:13:20.385237160 (14h33m19.974582137s) [upload=15079]: Done uploading segments... Aug 8 00:13:20.385255485 (14h33m19.974600472s) [upload=15079]: Upload finished. err: <nil> Aug 8 00:14:06.785741109 (14h34m6.375086096s) [upload=15091]: Starting upload Aug 8 00:14:06.785899151 (14h34m6.375244138s) [upload=15091]: Done starting upload Aug 8 00:14:06.786002628 (14h34m6.375347615s) [upload=15091]: Uploading segments... Aug 8 00:14:06.786034078 (14h34m6.375379065s) [upload=15091]: Got next segment. Inline: true Aug 8 00:14:06.786037786 (14h34m6.375382773s) [upload=15091]: Waiting for error group managing segments... Aug 8 00:14:06.786073094 (14h34m6.375418081s) [upload=15091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:14:06.786078945 (14h34m6.375423922s) [upload=15091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:14:06.786081339 (14h34m6.375426326s) [upload=15091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:14:07.092788097 (14h34m6.682133084s) [upload=15091]: Done uploading segments... Aug 8 00:14:07.092803166 (14h34m6.682148153s) [upload=15091]: Upload finished. err: <nil> Duration: 1.820880902s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X...X..........X.......X...X............X....X.........] duration: 5.857026388s FAIL (2025-08-08 00:46:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837172821 (14h59m54.426517798s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837385949 (14h59m54.426730936s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837507802 (14h59m54.426852779s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837640145 (14h59m54.426985122s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837819459 (14h59m54.427164446s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.837962502 (14h59m54.427307479s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838112790 (14h59m54.427457767s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838238691 (14h59m54.427583678s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838393447 (14h59m54.427738434s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838490402 (14h59m54.427835389s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838660288 (14h59m54.428005275s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838681548 (14h59m54.428026525s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838866312 (14h59m54.428211299s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.838893975 (14h59m54.428238962s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:54.839056986 (14h59m54.428401974s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Uploading piece... Aug 8 00:39:55.935384249 (14h59m55.524729246s) [upload=15536]: Next returned nil segment Aug 8 00:39:55.935402815 (14h59m55.524747802s) [upload=15536]: Waiting for error group managing segments... Aug 8 00:39:56.226788010 (14h59m55.816132997s) [noise=true , address=64.49.13.27:4968 , piece_id=IGWGIQXA7XCKBWG6CESCXQH5KW2ZM6JTMBRBYKIFAEUUDMBFGZOA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.265710491 (14h59m55.855055478s) [noise=true , address=79.127.205.245:4235 , piece_id=HE52XPW7FKQX2IU4RQSBROB4PDAQKUFELPVJD3D3QBFWDKEG5URA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.326363980 (14h59m55.915708967s) [noise=true , address=64.49.13.169:4709 , piece_id=PIPH24KH4WITXCTG5KI26BQSCGWYJ5ABGWOCSA3VDEBWEMNY5FTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773039652 (14h59m56.362384679s) [noise=true , address=45.91.116.13:4187 , piece_id=GX5YWIWJK6AM256TG4JMBTL7MM7C5EOHRMUAMQISZNHQBBBUVDTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.773213735 (14h59m56.362558732s) [noise=true , address=104.36.82.15:4292 , piece_id=X6JRJ3RQHG5AJBDUDA44JV74VZZUZZ72MPOXBTVLC4MVMS4D5ESQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.782851711 (14h59m56.372196698s) [noise=true , address=104.36.82.8:4989 , piece_id=KOO4JA6L6TAXIOMDMC3425A4KHG2MK2OHY2KVFLAUMPCX3M26O5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854376586 (14h59m56.443721573s) [noise=true , address=104.36.82.46:4484 , piece_id=BMHSO2DIXVRRHUZRHWUQ24TXMMTRMVX34SVDUUUMRMKPSUQG3APQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.854703231 (14h59m56.444048218s) [noise=true , address=104.36.82.17:4337 , piece_id=N6F66L3YEZ57UDLAFXE452G35SLUJV63QCTXM6CKCY3KK7JIAI5Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.914818370 (14h59m56.504163357s) [noise=true , address=104.36.82.8:4981 , piece_id=3WXK233GLJ2HGSIAW7CXFTEXDTIWK2KC2CO73JIRGKDE5YL4QKWQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.915843802 (14h59m56.505188849s) [noise=true , address=104.36.82.35:4616 , piece_id=SGA55FQF2EQZPXVOWDFAK7BFCNIXXT3U2QXYNWSOLMCI33JQJNTA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.942005621 (14h59m56.531350608s) [noise=true , address=104.36.82.34:4577 , piece_id=SL5HJHSPSJCJLBFWP6VRTIXJJDAWXFT65VM34KYMHDCHSYST6KBA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:56.981103286 (14h59m56.570448273s) [noise=true , address=104.36.82.43:4857 , piece_id=OLV5MYZ3OG27WFA6OMQA645VVCTCIWUEOVRBCZKEZF463DRFJC3A, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.095564053 (14h59m56.68490904s) [noise=true , address=104.36.82.56:4229 , piece_id=RZS5K5CYPGWVCRBNQX4SOAVS3E3TBUPDM5RLP6R3CV522VDEE63Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216182613 (14h59m56.8055276s) [noise=true , address=104.36.82.62:4439 , piece_id=OPCFN362QAB3SMGMUUW4UKEAG2BZGI55L4K46RBAM4BSQOGCTV2Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: <nil> Aug 8 00:39:57.216206869 (14h59m56.805551856s) [seg_pos={0 0}, upload=15536]: Segment reached optimal threshold of 14 pieces. Aug 8 00:39:57.216406632 (14h59m56.805751619s) [noise=true , address=104.36.82.59:4321 , piece_id=S7OKLOZO2QMHL33GSU4OCDURGTK6DUES5342I5CBNCTT32WKJFKA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bN25eEEjxCWKJSr8qj7moDJxArgfsjzPTdBcLdfvoeyxiqPpc): protocol: expected piece hash; context canceled; EOF Aug 8 00:39:57.216460465 (14h59m56.805805452s) [noise=true , address=104.36.82.5:4869 , piece_id=JBCDCG676BBV6U2GAVWDQIWYXKKQGW2DKK6MDY74LNPGP4K3CTQA, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38660->104.36.82.5:4869: use of closed network connection Aug 8 00:39:57.216644467 (14h59m56.805989454s) [noise=true , address=104.36.82.47:4521 , piece_id=OAZDQBO65EYCY3DMYHXW2INHK3K5WARUFEM66NZSXOO7B6JTKZCQ, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD): protocol: read tcp 188.166.225.170:50728->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216665567 (14h59m56.806010554s) [noise=true , address=104.36.82.61:4383 , piece_id=3ESRNG4ZVAEXL5EKL3VJKDZNCM6SBEUN4WDDDPL6J4YRD5V47P6Q, seg_pos={0 0}, upload=15536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: read tcp 188.166.225.170:45052->104.36.82.61:4383: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:39:57.216679534 (14h59m56.806024521s) [seg_pos={0 0}, upload=15536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:39:57.216739699 (14h59m56.806084686s) [upload=15536]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:39:57.216743286 (14h59m56.806088273s) [upload=15536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:39:57.516798834 (14h59m57.106143821s) [upload=15536]: Done uploading segments... Aug 8 00:39:57.516817129 (14h59m57.106162116s) [upload=15536]: Upload finished. err: <nil> Aug 8 00:40:09.579830255 (15h0m9.169175242s) [upload=15541]: Starting upload Aug 8 00:40:09.579877025 (15h0m9.169222012s) [upload=15541]: Done starting upload Aug 8 00:40:09.579941739 (15h0m9.169286726s) [upload=15541]: Uploading segments... Aug 8 00:40:09.580164405 (15h0m9.169509432s) [upload=15541]: Got next segment. Inline: false Aug 8 00:40:09.580219460 (15h0m9.169564477s) [upload=15541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:40:09.580251472 (15h0m9.169596489s) [upload=15541]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:40:09.797368422 (15h0m9.386713409s) [upload=15541]: Next segment err: metaclient: Too Many Requests Aug 8 00:40:09.797398149 (15h0m9.386743136s) [upload=15541]: Done uploading segments... Aug 8 00:40:09.797473614 (15h0m9.386818601s) [upload=15541]: Upload finished. err: metaclient: Too Many Requests Duration: 5.857048721s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.990342853s FAIL (2025-08-08 00:46:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3.700898256s) [upload=14855]: Done starting upload Aug 8 00:00:34.111602513 (14h20m33.7009475s) [upload=14855]: Uploading segments... Aug 8 00:00:34.111774201 (14h20m33.701119218s) [upload=14855]: Got next segment. Inline: false Aug 8 00:00:34.111816783 (14h20m33.70116178s) [upload=14855]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:34.111822464 (14h20m33.701167461s) [upload=14855]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:01:03.465979471 (14h21m3.055324468s) [upload=14855]: Next segment err: context deadline exceeded Aug 8 00:01:03.466148254 (14h21m3.055493251s) [upload=14855]: Done uploading segments... Aug 8 00:01:03.466174945 (14h21m3.055519932s) [upload=14855]: Upload finished. err: context deadline exceeded Aug 8 00:01:19.453976971 (14h21m19.043321958s) [upload=14872]: Starting upload Aug 8 00:01:19.454042106 (14h21m19.043387123s) [upload=14872]: Done starting upload Aug 8 00:01:19.454156104 (14h21m19.043501091s) [upload=14872]: Uploading segments... Aug 8 00:01:19.454244503 (14h21m19.04358949s) [upload=14872]: Got next segment. Inline: false Aug 8 00:01:19.454272567 (14h21m19.043617554s) [upload=14872]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:19.454276825 (14h21m19.043621812s) [upload=14872]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:01:39.582519114 (14h21m39.171864101s) [seg_pos={0 0}, upload=14872]: Begin upload segment... Aug 8 00:01:39.582565882 (14h21m39.17191087s) [seg_pos={0 0}, upload=14872]: Done begin upload segment. Aug 8 00:01:39.582581332 (14h21m39.17192632s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.582590870 (14h21m39.171935857s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.582951871 (14h21m39.172296858s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583059888 (14h21m39.172404905s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583214905 (14h21m39.172559892s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583427882 (14h21m39.172772869s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583689032 (14h21m39.173034019s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583925344 (14h21m39.173270331s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584272317 (14h21m39.173617294s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584579135 (14h21m39.173924122s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584852117 (14h21m39.174197104s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585105202 (14h21m39.174450179s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585298762 (14h21m39.174643739s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585506369 (14h21m39.174851346s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585763882 (14h21m39.175108869s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.586021575 (14h21m39.175366562s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:41.370103161 (14h21m40.959448138s) [upload=14872]: Next returned nil segment Aug 8 00:01:41.370130243 (14h21m40.95947523s) [upload=14872]: Waiting for error group managing segments... Aug 8 00:01:41.673384945 (14h21m41.262729922s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.739890780 (14h21m41.329235767s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.743845828 (14h21m41.333190815s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.786086095 (14h21m41.375431072s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.791525722 (14h21m41.380870699s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.959657137 (14h21m41.549002124s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.000743686 (14h21m41.590088693s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.160166408 (14h21m41.749511395s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.175399653 (14h21m41.764744641s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.175880162 (14h21m41.765225139s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.278394081 (14h21m41.867739068s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.293663936 (14h21m41.883008933s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.309113927 (14h21m41.898458904s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.309137942 (14h21m41.898482919s) [seg_pos={0 0}, upload=14872]: Segment reached optimal threshold of 13 pieces. Aug 8 00:01:42.309315973 (14h21m41.89866099s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:42.309402328 (14h21m41.898747305s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:42.309425222 (14h21m41.898770229s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection Aug 8 00:01:42.309450360 (14h21m41.898795347s) [seg_pos={0 0}, upload=14872]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 8 00:01:42.309540663 (14h21m41.89888565s) [upload=14872]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:01:42.309548688 (14h21m41.898893735s) [upload=14872]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:57.644545293 (14h21m57.2338903s) [upload=14872]: Done uploading segments... Aug 8 00:01:57.644581281 (14h21m57.233926288s) [upload=14872]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.990367339s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...X.......................................................] duration: 6.629894093s FAIL (2025-08-08 00:46:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.246137011 (14h21m15.835481998s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.246560380 (14h21m15.835905367s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247408974 (14h21m15.836753961s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247678410 (14h21m15.837023397s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247919882 (14h21m15.837264869s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247943888 (14h21m15.837288875s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248293266 (14h21m15.837638253s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248420540 (14h21m15.837765527s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248557311 (14h21m15.837902288s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248758506 (14h21m15.838103493s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248897792 (14h21m15.838242779s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248989629 (14h21m15.838334616s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249186575 (14h21m15.838531552s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249273772 (14h21m15.838618759s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249459598 (14h21m15.838804585s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249717240 (14h21m15.839062227s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:17.413257645 (14h21m17.002602632s) [upload=14870]: Next returned nil segment Aug 8 00:01:17.413273324 (14h21m17.002618311s) [upload=14870]: Waiting for error group managing segments... Aug 8 00:01:17.710897294 (14h21m17.300242281s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:17.715058896 (14h21m17.304403883s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.126218161 (14h21m17.715563148s) [noise=true , address=79.127.201.215:4356 , piece_id=VWNEIHNI4KPPF4QLW36INIWTUI2CD6MZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.139272367 (14h21m17.728617354s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.221269031 (14h21m17.810614018s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.257350144 (14h21m17.846695121s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.258435089 (14h21m17.847780076s) [noise=true , address=104.36.82.44:4876 , piece_id=BA2G42OXQD3NLONUVD475S2BNC7T37T5PHPE3K3E5HJAJP4W5MKQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.287010923 (14h21m17.87635591s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.348793001 (14h21m17.938137988s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.378129459 (14h21m17.967474436s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.386414901 (14h21m17.975759888s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.516150823 (14h21m18.10549581s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.626647204 (14h21m18.215992191s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:19.042108811 (14h21m18.631453798s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:19.042136654 (14h21m18.631481641s) [seg_pos={0 0}, upload=14870]: Segment reached optimal threshold of 14 pieces. Aug 8 00:01:19.042375541 (14h21m18.631720518s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:19.042485912 (14h21m18.6318309s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: rpc: dial tcp 104.36.82.55:4209: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:01:19.042532361 (14h21m18.631877348s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqNLPRWY7fJRS6F9h4B4qVFv6U5jyWkCy3Um7jfjEdCShEjsD): protocol: write tcp 188.166.225.170:37056->104.36.82.20:4466: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:01:19.042616462 (14h21m18.631961449s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa): protocol: rpc: dial tcp 104.36.82.5:4844: operation was canceled; conn closed: op; piecestore: piecestore close: context canceled Aug 8 00:01:19.042628876 (14h21m18.631973863s) [seg_pos={0 0}, upload=14870]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:01:19.042699029 (14h21m18.632044017s) [upload=14870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:01:19.042703128 (14h21m18.632048115s) [upload=14870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:19.344335316 (14h21m18.933680293s) [upload=14870]: Done uploading segments... Aug 8 00:01:19.344353089 (14h21m18.933698077s) [upload=14870]: Upload finished. err: <nil> Aug 8 00:02:30.778225454 (14h22m30.367570441s) [upload=14892]: Starting upload Aug 8 00:02:30.778283114 (14h22m30.367628151s) [upload=14892]: Done starting upload Aug 8 00:02:30.778411841 (14h22m30.367756828s) [upload=14892]: Uploading segments... Aug 8 00:02:30.778539616 (14h22m30.367884623s) [upload=14892]: Got next segment. Inline: false Aug 8 00:02:30.778583389 (14h22m30.367928376s) [upload=14892]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:30.778587717 (14h22m30.367932704s) [upload=14892]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:03:30.135980490 (14h23m29.725325467s) [upload=14892]: Next segment err: context deadline exceeded Aug 8 00:03:30.136177137 (14h23m29.725522124s) [upload=14892]: Done uploading segments... Aug 8 00:03:30.136196834 (14h23m29.725541821s) [upload=14892]: Upload finished. err: context deadline exceeded Duration: 6.62992322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [.XXXX....X..................................................] duration: 1.869854003s FAIL (2025-08-08 00:46:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9109253s) [upload=14753]: Starting upload Aug 7 23:54:04.489828037 (14h14m4.079173014s) [upload=14753]: Done starting upload Aug 7 23:54:04.489877612 (14h14m4.079222589s) [upload=14753]: Uploading segments... Aug 7 23:54:04.489903933 (14h14m4.07924892s) [upload=14753]: Got next segment. Inline: true Aug 7 23:54:04.489907469 (14h14m4.079252446s) [upload=14753]: Waiting for error group managing segments... Aug 7 23:54:04.489939651 (14h14m4.079284638s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:04.489943298 (14h14m4.079288285s) [upload=14753]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:04.489945472 (14h14m4.079290459s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:04.778726062 (14h14m4.368071049s) [upload=14753]: Done uploading segments... Aug 7 23:54:04.778744989 (14h14m4.368089976s) [upload=14753]: Upload finished. err: <nil> Aug 7 23:55:14.199719855 (14h15m13.789064892s) [upload=14770]: Starting upload Aug 7 23:55:14.199777645 (14h15m13.789122632s) [upload=14770]: Done starting upload Aug 7 23:55:14.199824415 (14h15m13.789169392s) [upload=14770]: Uploading segments... Aug 7 23:55:14.199852819 (14h15m13.789197806s) [upload=14770]: Got next segment. Inline: true Aug 7 23:55:14.199856436 (14h15m13.789201423s) [upload=14770]: Waiting for error group managing segments... Aug 7 23:55:14.199889419 (14h15m13.789234406s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:14.199893557 (14h15m13.789238544s) [upload=14770]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:14.199895951 (14h15m13.789240938s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:13.542516296 (14h16m13.131861283s) [upload=14770]: Done uploading segments... Aug 7 23:56:13.542536334 (14h16m13.131881321s) [upload=14770]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:14.428677736 (14h16m14.018022723s) [upload=14788]: Starting upload Aug 7 23:56:14.428730978 (14h16m14.018075965s) [upload=14788]: Done starting upload Aug 7 23:56:14.428789289 (14h16m14.018134276s) [upload=14788]: Uploading segments... Aug 7 23:56:14.428819026 (14h16m14.018164013s) [upload=14788]: Got next segment. Inline: true Aug 7 23:56:14.428822873 (14h16m14.01816785s) [upload=14788]: Waiting for error group managing segments... Aug 7 23:56:14.428855215 (14h16m14.018200202s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:14.428858562 (14h16m14.018203549s) [upload=14788]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:14.428860736 (14h16m14.018205723s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:13.781984834 (14h17m13.371329821s) [upload=14788]: Done uploading segments... Aug 7 23:57:13.782003370 (14h17m13.371348357s) [upload=14788]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:29.079359478 (14h17m28.668704466s) [upload=14808]: Starting upload Aug 7 23:57:29.079416147 (14h17m28.668761134s) [upload=14808]: Done starting upload Aug 7 23:57:29.079461845 (14h17m28.668806822s) [upload=14808]: Uploading segments... Aug 7 23:57:29.079488876 (14h17m28.668833863s) [upload=14808]: Got next segment. Inline: true Aug 7 23:57:29.079492263 (14h17m28.66883725s) [upload=14808]: Waiting for error group managing segments... Aug 7 23:57:29.079523392 (14h17m28.668868379s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:29.079527049 (14h17m28.668872036s) [upload=14808]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:29.079529484 (14h17m28.668874461s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:28.439377526 (14h18m28.028722513s) [upload=14808]: Done uploading segments... Aug 7 23:58:28.439395721 (14h18m28.028740708s) [upload=14808]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:29.333669502 (14h18m28.923014489s) [upload=14825]: Starting upload Aug 7 23:58:29.333742581 (14h18m28.923087568s) [upload=14825]: Done starting upload Aug 7 23:58:29.333835740 (14h18m28.923180727s) [upload=14825]: Uploading segments... Aug 7 23:58:29.333878000 (14h18m28.923222987s) [upload=14825]: Got next segment. Inline: true Aug 7 23:58:29.333881998 (14h18m28.923226975s) [upload=14825]: Waiting for error group managing segments... Aug 7 23:58:29.333911455 (14h18m28.923256442s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:29.333917496 (14h18m28.923262473s) [upload=14825]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:29.333919741 (14h18m28.923264728s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:28.692415040 (14h19m28.281760027s) [upload=14825]: Done uploading segments... Aug 7 23:59:28.692444777 (14h19m28.281789764s) [upload=14825]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:29.562470053 (14h19m29.15181504s) [upload=14840]: Starting upload Aug 7 23:59:29.562538985 (14h19m29.151883972s) [upload=14840]: Done starting upload Aug 7 23:59:29.562614629 (14h19m29.151959616s) [upload=14840]: Uploading segments... Aug 7 23:59:29.562650589 (14h19m29.151995566s) [upload=14840]: Got next segment. Inline: true Aug 7 23:59:29.562655498 (14h19m29.152000485s) [upload=14840]: Waiting for error group managing segments... Aug 7 23:59:29.562686507 (14h19m29.152031494s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:29.562690264 (14h19m29.152035241s) [upload=14840]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:29.562692428 (14h19m29.152037405s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:29.822497550 (14h19m29.411842537s) [upload=14840]: Done uploading segments... Aug 7 23:59:29.822514693 (14h19m29.41185968s) [upload=14840]: Upload finished. err: <nil> Aug 8 00:00:35.820554363 (14h20m35.40989935s) [upload=14856]: Starting upload Aug 8 00:00:35.820610611 (14h20m35.409955598s) [upload=14856]: Done starting upload Aug 8 00:00:35.820669503 (14h20m35.4100145s) [upload=14856]: Uploading segments... Aug 8 00:00:35.820742333 (14h20m35.41008732s) [upload=14856]: Got next segment. Inline: true Aug 8 00:00:35.820747282 (14h20m35.410092269s) [upload=14856]: Waiting for error group managing segments... Aug 8 00:00:35.820788371 (14h20m35.410133358s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.820793050 (14h20m35.410138037s) [upload=14856]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.820795485 (14h20m35.410140472s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:36.084321816 (14h20m35.673666803s) [upload=14856]: Done uploading segments... Aug 8 00:00:36.084338618 (14h20m35.673683615s) [upload=14856]: Upload finished. err: <nil> Aug 8 00:01:14.496466774 (14h21m14.085811781s) [upload=14868]: Starting upload Aug 8 00:01:14.496556174 (14h21m14.085901162s) [upload=14868]: Done starting upload Aug 8 00:01:14.496609206 (14h21m14.085954183s) [upload=14868]: Uploading segments... Aug 8 00:01:14.496639985 (14h21m14.085984972s) [upload=14868]: Got next segment. Inline: true Aug 8 00:01:14.496643592 (14h21m14.085988579s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:01:14.496678638 (14h21m14.086023625s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:14.496682386 (14h21m14.086027373s) [upload=14868]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:14.496684681 (14h21m14.086029658s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:14.760681029 (14h21m14.350026016s) [upload=14868]: Done uploading segments... Aug 8 00:01:14.760702580 (14h21m14.350047567s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:01:50.577049881 (14h21m50.166394858s) [upload=14878]: Starting upload Aug 8 00:01:50.577100538 (14h21m50.166445525s) [upload=14878]: Done starting upload Aug 8 00:01:50.577180091 (14h21m50.166525078s) [upload=14878]: Uploading segments... Aug 8 00:01:50.577228012 (14h21m50.166573s) [upload=14878]: Got next segment. Inline: true Aug 8 00:01:50.577231469 (14h21m50.166576456s) [upload=14878]: Waiting for error group managing segments... Aug 8 00:01:50.577270633 (14h21m50.166615621s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:50.577274721 (14h21m50.166619708s) [upload=14878]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:50.577276906 (14h21m50.166621893s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:50.863237339 (14h21m50.452582326s) [upload=14878]: Done uploading segments... Aug 8 00:01:50.863254973 (14h21m50.452599961s) [upload=14878]: Upload finished. err: <nil> Aug 8 00:02:41.474648957 (14h22m41.063993944s) [upload=14899]: Starting upload Aug 8 00:02:41.474716297 (14h22m41.064061274s) [upload=14899]: Done starting upload Aug 8 00:02:41.474771743 (14h22m41.06411673s) [upload=14899]: Uploading segments... Aug 8 00:02:41.474801099 (14h22m41.064146086s) [upload=14899]: Got next segment. Inline: true Aug 8 00:02:41.474804966 (14h22m41.064149953s) [upload=14899]: Waiting for error group managing segments... Aug 8 00:02:41.474841336 (14h22m41.064186323s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:41.474845214 (14h22m41.064190201s) [upload=14899]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:41.474847558 (14h22m41.064192535s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:03:40.835731182 (14h23m40.425076189s) [upload=14899]: Done uploading segments... Aug 8 00:03:40.835778212 (14h23m40.425123199s) [upload=14899]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.869879381s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.640191408s FAIL (2025-08-08 00:46:41 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.640214423s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 6 out of 60 is failed (threshold=3) [X.XXXX....X.................................................] duration: 1.91736849s FAIL (2025-08-08 00:46:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9109253s) [upload=14753]: Starting upload Aug 7 23:54:04.489828037 (14h14m4.079173014s) [upload=14753]: Done starting upload Aug 7 23:54:04.489877612 (14h14m4.079222589s) [upload=14753]: Uploading segments... Aug 7 23:54:04.489903933 (14h14m4.07924892s) [upload=14753]: Got next segment. Inline: true Aug 7 23:54:04.489907469 (14h14m4.079252446s) [upload=14753]: Waiting for error group managing segments... Aug 7 23:54:04.489939651 (14h14m4.079284638s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:04.489943298 (14h14m4.079288285s) [upload=14753]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:04.489945472 (14h14m4.079290459s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:04.778726062 (14h14m4.368071049s) [upload=14753]: Done uploading segments... Aug 7 23:54:04.778744989 (14h14m4.368089976s) [upload=14753]: Upload finished. err: <nil> Aug 7 23:55:14.199719855 (14h15m13.789064892s) [upload=14770]: Starting upload Aug 7 23:55:14.199777645 (14h15m13.789122632s) [upload=14770]: Done starting upload Aug 7 23:55:14.199824415 (14h15m13.789169392s) [upload=14770]: Uploading segments... Aug 7 23:55:14.199852819 (14h15m13.789197806s) [upload=14770]: Got next segment. Inline: true Aug 7 23:55:14.199856436 (14h15m13.789201423s) [upload=14770]: Waiting for error group managing segments... Aug 7 23:55:14.199889419 (14h15m13.789234406s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:14.199893557 (14h15m13.789238544s) [upload=14770]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:14.199895951 (14h15m13.789240938s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:13.542516296 (14h16m13.131861283s) [upload=14770]: Done uploading segments... Aug 7 23:56:13.542536334 (14h16m13.131881321s) [upload=14770]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:14.428677736 (14h16m14.018022723s) [upload=14788]: Starting upload Aug 7 23:56:14.428730978 (14h16m14.018075965s) [upload=14788]: Done starting upload Aug 7 23:56:14.428789289 (14h16m14.018134276s) [upload=14788]: Uploading segments... Aug 7 23:56:14.428819026 (14h16m14.018164013s) [upload=14788]: Got next segment. Inline: true Aug 7 23:56:14.428822873 (14h16m14.01816785s) [upload=14788]: Waiting for error group managing segments... Aug 7 23:56:14.428855215 (14h16m14.018200202s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:14.428858562 (14h16m14.018203549s) [upload=14788]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:14.428860736 (14h16m14.018205723s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:13.781984834 (14h17m13.371329821s) [upload=14788]: Done uploading segments... Aug 7 23:57:13.782003370 (14h17m13.371348357s) [upload=14788]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:29.079359478 (14h17m28.668704466s) [upload=14808]: Starting upload Aug 7 23:57:29.079416147 (14h17m28.668761134s) [upload=14808]: Done starting upload Aug 7 23:57:29.079461845 (14h17m28.668806822s) [upload=14808]: Uploading segments... Aug 7 23:57:29.079488876 (14h17m28.668833863s) [upload=14808]: Got next segment. Inline: true Aug 7 23:57:29.079492263 (14h17m28.66883725s) [upload=14808]: Waiting for error group managing segments... Aug 7 23:57:29.079523392 (14h17m28.668868379s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:29.079527049 (14h17m28.668872036s) [upload=14808]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:29.079529484 (14h17m28.668874461s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:28.439377526 (14h18m28.028722513s) [upload=14808]: Done uploading segments... Aug 7 23:58:28.439395721 (14h18m28.028740708s) [upload=14808]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:29.333669502 (14h18m28.923014489s) [upload=14825]: Starting upload Aug 7 23:58:29.333742581 (14h18m28.923087568s) [upload=14825]: Done starting upload Aug 7 23:58:29.333835740 (14h18m28.923180727s) [upload=14825]: Uploading segments... Aug 7 23:58:29.333878000 (14h18m28.923222987s) [upload=14825]: Got next segment. Inline: true Aug 7 23:58:29.333881998 (14h18m28.923226975s) [upload=14825]: Waiting for error group managing segments... Aug 7 23:58:29.333911455 (14h18m28.923256442s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:29.333917496 (14h18m28.923262473s) [upload=14825]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:29.333919741 (14h18m28.923264728s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:28.692415040 (14h19m28.281760027s) [upload=14825]: Done uploading segments... Aug 7 23:59:28.692444777 (14h19m28.281789764s) [upload=14825]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:29.562470053 (14h19m29.15181504s) [upload=14840]: Starting upload Aug 7 23:59:29.562538985 (14h19m29.151883972s) [upload=14840]: Done starting upload Aug 7 23:59:29.562614629 (14h19m29.151959616s) [upload=14840]: Uploading segments... Aug 7 23:59:29.562650589 (14h19m29.151995566s) [upload=14840]: Got next segment. Inline: true Aug 7 23:59:29.562655498 (14h19m29.152000485s) [upload=14840]: Waiting for error group managing segments... Aug 7 23:59:29.562686507 (14h19m29.152031494s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:29.562690264 (14h19m29.152035241s) [upload=14840]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:29.562692428 (14h19m29.152037405s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:29.822497550 (14h19m29.411842537s) [upload=14840]: Done uploading segments... Aug 7 23:59:29.822514693 (14h19m29.41185968s) [upload=14840]: Upload finished. err: <nil> Aug 8 00:00:35.820554363 (14h20m35.40989935s) [upload=14856]: Starting upload Aug 8 00:00:35.820610611 (14h20m35.409955598s) [upload=14856]: Done starting upload Aug 8 00:00:35.820669503 (14h20m35.4100145s) [upload=14856]: Uploading segments... Aug 8 00:00:35.820742333 (14h20m35.41008732s) [upload=14856]: Got next segment. Inline: true Aug 8 00:00:35.820747282 (14h20m35.410092269s) [upload=14856]: Waiting for error group managing segments... Aug 8 00:00:35.820788371 (14h20m35.410133358s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.820793050 (14h20m35.410138037s) [upload=14856]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.820795485 (14h20m35.410140472s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:36.084321816 (14h20m35.673666803s) [upload=14856]: Done uploading segments... Aug 8 00:00:36.084338618 (14h20m35.673683615s) [upload=14856]: Upload finished. err: <nil> Aug 8 00:01:14.496466774 (14h21m14.085811781s) [upload=14868]: Starting upload Aug 8 00:01:14.496556174 (14h21m14.085901162s) [upload=14868]: Done starting upload Aug 8 00:01:14.496609206 (14h21m14.085954183s) [upload=14868]: Uploading segments... Aug 8 00:01:14.496639985 (14h21m14.085984972s) [upload=14868]: Got next segment. Inline: true Aug 8 00:01:14.496643592 (14h21m14.085988579s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:01:14.496678638 (14h21m14.086023625s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:14.496682386 (14h21m14.086027373s) [upload=14868]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:14.496684681 (14h21m14.086029658s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:14.760681029 (14h21m14.350026016s) [upload=14868]: Done uploading segments... Aug 8 00:01:14.760702580 (14h21m14.350047567s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:01:50.577049881 (14h21m50.166394858s) [upload=14878]: Starting upload Aug 8 00:01:50.577100538 (14h21m50.166445525s) [upload=14878]: Done starting upload Aug 8 00:01:50.577180091 (14h21m50.166525078s) [upload=14878]: Uploading segments... Aug 8 00:01:50.577228012 (14h21m50.166573s) [upload=14878]: Got next segment. Inline: true Aug 8 00:01:50.577231469 (14h21m50.166576456s) [upload=14878]: Waiting for error group managing segments... Aug 8 00:01:50.577270633 (14h21m50.166615621s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:50.577274721 (14h21m50.166619708s) [upload=14878]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:50.577276906 (14h21m50.166621893s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:50.863237339 (14h21m50.452582326s) [upload=14878]: Done uploading segments... Aug 8 00:01:50.863254973 (14h21m50.452599961s) [upload=14878]: Upload finished. err: <nil> Aug 8 00:02:41.474648957 (14h22m41.063993944s) [upload=14899]: Starting upload Aug 8 00:02:41.474716297 (14h22m41.064061274s) [upload=14899]: Done starting upload Aug 8 00:02:41.474771743 (14h22m41.06411673s) [upload=14899]: Uploading segments... Aug 8 00:02:41.474801099 (14h22m41.064146086s) [upload=14899]: Got next segment. Inline: true Aug 8 00:02:41.474804966 (14h22m41.064149953s) [upload=14899]: Waiting for error group managing segments... Aug 8 00:02:41.474841336 (14h22m41.064186323s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:41.474845214 (14h22m41.064190201s) [upload=14899]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:41.474847558 (14h22m41.064192535s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:03:40.835731182 (14h23m40.425076189s) [upload=14899]: Done uploading segments... Aug 8 00:03:40.835778212 (14h23m40.425123199s) [upload=14899]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.917396684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [XX...X......................................................] duration: 6.088765084s FAIL (2025-08-08 00:46:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.246137011 (14h21m15.835481998s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.246560380 (14h21m15.835905367s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247408974 (14h21m15.836753961s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247678410 (14h21m15.837023397s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247919882 (14h21m15.837264869s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247943888 (14h21m15.837288875s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248293266 (14h21m15.837638253s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248420540 (14h21m15.837765527s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248557311 (14h21m15.837902288s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248758506 (14h21m15.838103493s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248897792 (14h21m15.838242779s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248989629 (14h21m15.838334616s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249186575 (14h21m15.838531552s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249273772 (14h21m15.838618759s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249459598 (14h21m15.838804585s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249717240 (14h21m15.839062227s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:17.413257645 (14h21m17.002602632s) [upload=14870]: Next returned nil segment Aug 8 00:01:17.413273324 (14h21m17.002618311s) [upload=14870]: Waiting for error group managing segments... Aug 8 00:01:17.710897294 (14h21m17.300242281s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:17.715058896 (14h21m17.304403883s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.126218161 (14h21m17.715563148s) [noise=true , address=79.127.201.215:4356 , piece_id=VWNEIHNI4KPPF4QLW36INIWTUI2CD6MZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.139272367 (14h21m17.728617354s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.221269031 (14h21m17.810614018s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.257350144 (14h21m17.846695121s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.258435089 (14h21m17.847780076s) [noise=true , address=104.36.82.44:4876 , piece_id=BA2G42OXQD3NLONUVD475S2BNC7T37T5PHPE3K3E5HJAJP4W5MKQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.287010923 (14h21m17.87635591s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.348793001 (14h21m17.938137988s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.378129459 (14h21m17.967474436s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.386414901 (14h21m17.975759888s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.516150823 (14h21m18.10549581s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.626647204 (14h21m18.215992191s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:19.042108811 (14h21m18.631453798s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:19.042136654 (14h21m18.631481641s) [seg_pos={0 0}, upload=14870]: Segment reached optimal threshold of 14 pieces. Aug 8 00:01:19.042375541 (14h21m18.631720518s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:19.042485912 (14h21m18.6318309s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: rpc: dial tcp 104.36.82.55:4209: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:01:19.042532361 (14h21m18.631877348s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqNLPRWY7fJRS6F9h4B4qVFv6U5jyWkCy3Um7jfjEdCShEjsD): protocol: write tcp 188.166.225.170:37056->104.36.82.20:4466: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:01:19.042616462 (14h21m18.631961449s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa): protocol: rpc: dial tcp 104.36.82.5:4844: operation was canceled; conn closed: op; piecestore: piecestore close: context canceled Aug 8 00:01:19.042628876 (14h21m18.631973863s) [seg_pos={0 0}, upload=14870]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:01:19.042699029 (14h21m18.632044017s) [upload=14870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:01:19.042703128 (14h21m18.632048115s) [upload=14870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:19.344335316 (14h21m18.933680293s) [upload=14870]: Done uploading segments... Aug 8 00:01:19.344353089 (14h21m18.933698077s) [upload=14870]: Upload finished. err: <nil> Aug 8 00:02:30.778225454 (14h22m30.367570441s) [upload=14892]: Starting upload Aug 8 00:02:30.778283114 (14h22m30.367628151s) [upload=14892]: Done starting upload Aug 8 00:02:30.778411841 (14h22m30.367756828s) [upload=14892]: Uploading segments... Aug 8 00:02:30.778539616 (14h22m30.367884623s) [upload=14892]: Got next segment. Inline: false Aug 8 00:02:30.778583389 (14h22m30.367928376s) [upload=14892]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:30.778587717 (14h22m30.367932704s) [upload=14892]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:03:30.135980490 (14h23m29.725325467s) [upload=14892]: Next segment err: context deadline exceeded Aug 8 00:03:30.136177137 (14h23m29.725522124s) [upload=14892]: Done uploading segments... Aug 8 00:03:30.136196834 (14h23m29.725541821s) [upload=14892]: Upload finished. err: context deadline exceeded Duration: 6.088790163s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)